PostgreSQL, Technology

[PostgreSQL] Amazon RDS as A Master: How to Replicate from AWS

There are lots of ways to replicate PostgreSQL database including streaming replication, logical replication, trigger-based replication and so on. And sometimes you need all of them depending on you and your business needs. I can say that replication options in Postgres is somewhat limitless.

And this time, I am writing how to replicate and use Amazon RDS as the master and use another as the slaves.

Replicate PostgreSQL from Amazon RDS

Why would you do that? Why would anyone bring down their cloud servers? This time, my reason is our data analysts needs to –as the name suggests– analyze transactions data in their own machine, which is located in our own data center while the database needed is on the cloud. We don’t allow them to access production servers because, you know, production is scary stuff. That’s what we told them. :grimacing:

Here’s the preparation stuff:

  • PostgreSQL servers >= 9.1 in Amazon RDS, this is because we need session replication role
  • Server for Bucardo and Postgres destination
  • Bucardo, obviously

That would be all.

Bucardo can be installed anywhere and this time I put it in the same server where Postgres destination is located. First, we need to grab Bucardo dependencies consist of some Perl modules including DBIx::Safe, boolean, DBI, DBD::Pg, Test::Simple.

sudo apt-get install libdbd-pg-perl libdbix-safe-perl postgresql-plperl-9.6 postgresql-contrib-9.6

Once they’re done, we can continue to grab the source files and install to the system. Continue reading “[PostgreSQL] Amazon RDS as A Master: How to Replicate from AWS”


[PostgreSQL] Bucardo Replication Update is Slow?

Bucardo is one of the replication system for PostgreSQL where it works asynchronous and using triggers to replicate the data. And one of the cool feature is you can have multi masters as well as slaves. Since the replication is working on table level to replicate the data, we can use it to replicate partial data and of course DLL is not supported. There are many more features you can look here.

Recently, I have to setup replication from Amazon RDS on one of our busiest databases to another cloud services provider for some reasons. And the only way to do this is using Bucardo.

At first, the replication went flawless. Setting up replication is done with triggers created on the required tables and the data started replicating as expected. The initial data was only 100GB and counting fast. There is not much INSERT activity on this database, but there are lots of UPDATEs depending on the hours. Bucardo log shows this on first days:

(12794) [Thu May  4 00:41:14 2017] KID (db_sync2) Totals: deletes=2216 inserts=3882 conflicts=0
(12794) [Thu May  4 00:41:22 2017] KID (db_sync2) Totals: deletes=1186 inserts=3663 conflicts=0
(12794) [Thu May  4 00:41:29 2017] KID (db_sync2) Totals: deletes=2511 inserts=3565 conflicts=0
(12794) [Thu May  4 00:41:37 2017] KID (db_sync2) Totals: deletes=3425 inserts=5385 conflicts=0
(12794) [Thu May  4 00:41:38 2017] KID (db_sync2) Totals: deletes=2938 inserts=3675 conflicts=0

Continue reading “[PostgreSQL] Bucardo Replication Update is Slow?”