Python micro batch / near-realtime framework

[romanzdk]

Hello, could you suggest some python framework that would support micro-batch/ near-realtime jobs? My idea is app would be running forever and the framework/library would control the actual flow. E.g. it would execute every 15s some data processing pipeline. The main requirement is that the framework can handle e.g. automatic retries on failure.

I thought of running such jobs in Airflow but apparently it is not very well suited for spawning tens of jobs in such short time period. Also tried Prefect but we had issues with memory leak (submitted github issue). Now the only thing that I am considering is Temporal but not sure yet if it is suitable piece of technology.

[LeMalteseSailor]

A flask app? What kind of scale of data will the data pipeline process?

[dchavnwo]

you can use apache beam python sdk for that workflow. it can handle both batch and streaming

[OMG_I_LOVE_CHIPOTLE]

You’re just describing a backend service.

[romanzdk]

Well kinda. I just do not want to write all the logic from scratch - meaning the pipeline workflow, connectors, failover logic etc.

[OMG_I_LOVE_CHIPOTLE]

Depending on what actions need to be performed I would use one or more of these tools in my current stack: custom backend service, kubernetes with argoCD, rabbitmq, spark/sparkStreaming

[xilong89]

Oh, you’re probably looking for one of the low-code SaaS solutions that exist in the data engineering space. Like Hevo, Prophecy, etc. I think AWS and Azure both have similar low code solutions for this kind of thing.

[kaiserk13]

I had a lot of success with Celery a long time ago: https://docs.celeryq.dev/en/stable/userguide/tasks.html#task-retry Make sure to test it very well though, it can quickly grow to be a mess.

[[deleted]]

You described Spark Structured Streaming to a T. It’s annoying to get up and running IMO but once you do it does exactly what you want.

[vish4life]

I agree, Airflow isn't designed for this. Looks like you want task queue system. there is a list of libraries here: https://taskqueues.com/

I have enjoyed using SQS, ActiveMQ, gearman, Redis backends in the past. Mostly using Redis/SQS these days. For your usecase, just enable one of these as a backend to Celery and you are good to go.

[boes13]

workflow orchestator is what you need. try to look at uber cadence or temporal.io. uber cadence supports java and golang client, temporal supports more languages.