PostgreSQL, Technology

[PostgreSQL] Amazon RDS as A Master: How to Replicate from AWS

There are lots of ways to replicate PostgreSQL database including streaming replication, logical replication, trigger-based replication and so on. And sometimes you need all of them depending on you and your business needs. I can say that replication options in Postgres is somewhat limitless.

And this time, I am writing how to replicate and use Amazon RDS as the master and use another as the slaves.

Replicate PostgreSQL from Amazon RDS

Why would you do that? Why would anyone bring down their cloud servers? This time, my reason is our data analysts needs to –as the name suggests– analyze transactions data in their own machine, which is located in our own data center while the database needed is on the cloud. We don’t allow them to access production servers because, you know, production is scary stuff. That’s what we told them. :grimacing:

Here’s the preparation stuff:

  • PostgreSQL servers >= 9.1 in Amazon RDS, this is because we need session replication role
  • Server for Bucardo and Postgres destination
  • Bucardo, obviously

That would be all.

Bucardo can be installed anywhere and this time I put it in the same server where Postgres destination is located. First, we need to grab Bucardo dependencies consist of some Perl modules including DBIx::Safe, boolean, DBI, DBD::Pg, Test::Simple.

sudo apt-get install libdbd-pg-perl libdbix-safe-perl postgresql-plperl-9.6 postgresql-contrib-9.6

Once they’re done, we can continue to grab the source files and install to the system. Continue reading “[PostgreSQL] Amazon RDS as A Master: How to Replicate from AWS”

PostgreSQL

[PostgreSQL] Bucardo Replication Update is Slow?

Bucardo is one of the replication system for PostgreSQL where it works asynchronous and using triggers to replicate the data. And one of the cool feature is you can have multi masters as well as slaves. Since the replication is working on table level to replicate the data, we can use it to replicate partial data and of course DLL is not supported. There are many more features you can look here.

Recently, I have to setup replication from Amazon RDS on one of our busiest databases to another cloud services provider for some reasons. And the only way to do this is using Bucardo.

At first, the replication went flawless. Setting up replication is done with triggers created on the required tables and the data started replicating as expected. The initial data was only 100GB and counting fast. There is not much INSERT activity on this database, but there are lots of UPDATEs depending on the hours. Bucardo log shows this on first days:

(12794) [Thu May  4 00:41:14 2017] KID (db_sync2) Totals: deletes=2216 inserts=3882 conflicts=0
(12794) [Thu May  4 00:41:22 2017] KID (db_sync2) Totals: deletes=1186 inserts=3663 conflicts=0
(12794) [Thu May  4 00:41:29 2017] KID (db_sync2) Totals: deletes=2511 inserts=3565 conflicts=0
(12794) [Thu May  4 00:41:37 2017] KID (db_sync2) Totals: deletes=3425 inserts=5385 conflicts=0
(12794) [Thu May  4 00:41:38 2017] KID (db_sync2) Totals: deletes=2938 inserts=3675 conflicts=0

Continue reading “[PostgreSQL] Bucardo Replication Update is Slow?”

Miscellaneous

[NakaTrip] Mengarungi Sungai Cicatih (Citatih)

Terakhir kali berarung jeram adalah saat saya duduk di bangku kuliah. Saat itu saya aktif sebagai trainer outbond bersama beberapa kawan senior untuk bersenang-senang dan tentu saja dapat duit. Setelah beberapa tahun lamanya, akhirnya saya kembali mengarungi sungai bersama sebuah organizer bernama Riam Jeram.

NakaTrip 2017 - arung jeram di Sungai Cicatih

Namun, kali ini saya tidak akan menceritakan detail bagaimana perjalanan arung jeram yang kami lakukan pada 8 April kemarin, melainkan apa yang bisa disimpulkan supaya menjadi pelajaran bagi banyak orang, my takeaways. Maka, tanpa basa-basi…

Continue reading “[NakaTrip] Mengarungi Sungai Cicatih (Citatih)”

Miscellaneous, viewpoint

Mau ke Big Bad Wolf Books 2017?

Sebagai pencinta buku, hari Kamis 20 April 2017 lalu saya berkesempatan datang ke hari pertama (pembukaan) Big Bad Wolf Books 2017. Ini adalah kali pertama datang ke pameran buku yang besar ini dan saya harus akui bahwa datang ke sini ini membutuhkan perjuangan yang besar sekali. Dan ya, hasilnya juga lumayan.

Apa itu Big Bad Wolf Books? BBW adalah bazaar buku yang diselenggarakan oleh Big Bad Wolf Sendirian Berhad (Sdn Bhd) dan dimulai di Malaysia kemudian menjalar ke negara di Asia Tenggara lain termasuk Indonesia. Nah, buku-buku dalam BBW adalah buku-buku clearance sale dari BookXCess, si induk semang BBW Sdn Bhd dan juga beberapa distributor lainnya. Info lebih lanjut silakan merujuk ke sini.

Nah, dari pengalaman sehari tersebut beberapa hal berikut inilah pelajaran yang dapat saya simpulkan:

Continue reading “Mau ke Big Bad Wolf Books 2017?”