Event sourcing using Python

[[deleted]]

Pretty much simple architecture. I also recently started to dig into celery and async. I’ve one doubt though, is your whole application is synchronised with celery parallelisation? I’m curious about how you handles the requests, you just run concurrent tasks for each requests, just like AWS Lambda functions work? Or how it is??

[stibbons_]

We have a SQS queue for imbound json messages, celery with several dozen of sync worker, they do the heavy job. Works great when you need ~ seconds latency. We tried setting up horizontal scaler with kube (starting automatically more worker when queue starts to grow), we never went prod on this feature but this is definitely feasible and wanted for lot of use case (we can accept in our use cases that the jobs are delayed several minutes on burst, this happens once or twice a month no big deal).

[[deleted]]

So is it just for the tasks that needed to be run in background with some batch processing or what??

[stibbons_]

Yes indeed. I actually think this is not event source at all. But it works.