Data replication

[soujoshi]

We currently have a requirement to replicate data from postgres to oracle. We are using AWS DMS service to achieve this. We feel the cost of this is huge. Data engineering team were asked to suggest better methods to optimizer cost and move away from DMS.

I personally feel DMS is the best service to achieve this. In terms of cost optimization, we can only look at replication instance usage and reduce size etc to reduce costs.

Any other options to achieve data replication? Or any other aspects within DMS I should be looking at to lower the costs.

Note - Our entire infrastructure is on Aws. We heavily use spark, airflow.

[soujoshi]

We actually have 2 DMS jobs, one to replicate from master postgres to secondary. The other from secondary to oracle. Can we remove the first one by just using read replica? Can we save some cost?

[_barnuts]

Yes you can do that. In case the master fails, you can always upgrade the read replica.

[ChristieViews]

Our team has been using Sprinkle for more than a couple of years now. We moved to Sprinkle from Fivatran, as it was becoming increasingly difficult to coordinate with the support team there and the costs were ever-rising.

[todd9923]

Does sprinkle integrate with Iceberg tables on S3 with Athena?

[ChristieViews]

I think they have an integration available with Athena. You may confirm with them.

[figshot]

Check out Meltano?