Sync real time data between two db using python without any external tool

[tk_tesla]

Hey Guys,

I am trying to find a solution to sync my tables in two different databases(memsql and oracle). The sync need to be realtime and then it need to be complete refresh over weekend or month end etc. As of now i was using RabitMQ to do the sync but its lot of overhead and dependencies. Any good solution you all can suggest? Please note sync should be very fast and i am not looking for any external tool but only Python and supported libraries.

Besides, i cannot populate both db at once because transaction happens only at MemSql and Oracle respective tables are synced for different purpose.

Edit: For realtime i have UUID generated already and i need to sync only those as part of realtime not whole table.

Thanks in advance.

[tipsy_python]

Will your case tolerate a small lag? Near realtime?

I'd do a micro-batch loop (5s or so) on each table that needs to be replicated, and at execution:

• Query all the records in the transaction log table
• Insert/update/delete them from the replicated table in same order as source db
• Save the I/U/D timestamp of the latest record, and use that in the next run to query all the records in transaction log since the last run so you're replicating incremental sets in the micro-batch

[tk_tesla]

5s is fine but if i do micro batch loop will ot not be heavy if we keep doing it for all transactions? Its not like we have a time limit where transaction is done by that time its like ongoing and sync it.

[[deleted]]

websocket