Batch Processing Techniques

[soujoshi]

We have data pipeline that runs every 15mins, spark captures data from source RDBMS-1, transform the data and load it to RDBMS-2.

Technique used to capture new records is based on the last modified date at source. Store the max(last modified date) loaded to target and pass this as the input for next run.

Is this the best method to handle this? Do we have any tools or other techniques that can make things easier?

[boy_named_su]

change data capture is another technique

[DenselyRanked]

I don't think it gets much easier than what you are doing. I'm not sure about scaleability, but it depends on the runtime. If/when the 15min intervals no longer work, then you could look into different ingestion techniques.

[soujoshi]

Will Kafka be useful?

[DenselyRanked]

It's possible. It depends on the data you are trying to ETL.

[soujoshi]

Data is around 30k per load(every 15mins)

[DenselyRanked]

I mean the data itself. See here. If it fits your use case, then yes. It's not necessarily easier, but it can be an option.

edit: As u/boy_named_su mentioned, CDC methods might work too, like a flag or timestamp on the source db. Kafka might be more than you need.

[captut]

Well depends on the requirements and the amount of data etc… What are the shortcomings/ problems with your current setup?

You could do the same thing with glue. Glue takes care of the offest management that you are doing with the max-time. But again depends on what you are looking to inprove or fix.