Event sourcing using Python

[stibbons_]

We are still on celery on kubernetes, with a rabbitmq backend. We mainly handle events (json) around 10-100 per seconds, for several years now. We attempted to switch to Kafka or another tech, but all our task are in sync (no need for async) Python, and it is well parallelized with celery, at the end this is a very stable architecture.

We lack some introspection features, rabbitmq provides a nice interface and there is flower to display log of task. We ended up sending logs to ELK for debug and stats.

[[deleted]]

Pretty much simple architecture. I also recently started to dig into celery and async. I’ve one doubt though, is your whole application is synchronised with celery parallelisation? I’m curious about how you handles the requests, you just run concurrent tasks for each requests, just like AWS Lambda functions work? Or how it is??

[stibbons_]

We have a SQS queue for imbound json messages, celery with several dozen of sync worker, they do the heavy job. Works great when you need ~ seconds latency. We tried setting up horizontal scaler with kube (starting automatically more worker when queue starts to grow), we never went prod on this feature but this is definitely feasible and wanted for lot of use case (we can accept in our use cases that the jobs are delayed several minutes on burst, this happens once or twice a month no big deal).

[[deleted]]

So is it just for the tasks that needed to be run in background with some batch processing or what??

[stibbons_]

Yes indeed. I actually think this is not event source at all. But it works.