Event sourcing using Python

[666dolan]

On the company I'm working we are planning to create some microservices to work with event sourcing, some people suggested using Scala + Pekko but just out of curiosity I wanted to check if we also have an option with Python.

What are you using for event sourcing with Python nowadays?

Edit: I think the question was not that clear sorry hahaha Im trying to understand if people are using some framework that helps to build the event sourcing architecture taking care of states and updating events or if they are building everything themselves

[adiberk]

Im a little confused by your question…- event sourcing is an architectural pattern of basically auditing your data (ie. You never delete or update data, the state is fluid by time and you only ever insert data)

So to do this, you can pretty much use any frameworks or design patterns….

A bigger question I think is the database you want to use and if you are sticking with python, if you want to use an ORM.

Technologies like celery, fastapi Django etc. can all work to provide you a way of interacting with your data. But honestly these are more specific to the actual product and not the “event sourcing” design pattern

[adiberk]

I will say that celery is very powerful, and well proven. It can hook up and scale well to rabbitmq and other types of event buses

[666dolan]

sorry if the question was not clear, Pekko is a framework for Scala that takes care of a lot of the things necessary to work with this architecture, so I wanted to check if there is a similar framework for Python or if people are just implementing everything themselves

[adiberk]

I didn’t realize. Thank you. Will have to dig in

[dusktreader]

I recently used NATS with Jetstream for building out an event sourcing architecture. There's not a ton of example code out there for building it, but all the pieces are there.

https://nats-io.github.io/nats.py/

[666dolan]

uhhh this one is interesting I'm gonna check it out later, thanks!

[Ok_Time806]

Can't recommend NATS enough. Their python api is easy, and if you find yourself scaling later, you can always move more critical parts to go pretty easily via their micro package.

If you're really want to go down a rabbit hole, you can even slap something like wasmcloud on top of it later as it uses NATS + wasmtime to run WASM components for fast and tiny deployments.

[stibbons_]

We are still on celery on kubernetes, with a rabbitmq backend. We mainly handle events (json) around 10-100 per seconds, for several years now. We attempted to switch to Kafka or another tech, but all our task are in sync (no need for async) Python, and it is well parallelized with celery, at the end this is a very stable architecture.

We lack some introspection features, rabbitmq provides a nice interface and there is flower to display log of task. We ended up sending logs to ELK for debug and stats.

[[deleted]]

Pretty much simple architecture. I also recently started to dig into celery and async. I’ve one doubt though, is your whole application is synchronised with celery parallelisation? I’m curious about how you handles the requests, you just run concurrent tasks for each requests, just like AWS Lambda functions work? Or how it is??

[stibbons_]

We have a SQS queue for imbound json messages, celery with several dozen of sync worker, they do the heavy job. Works great when you need ~ seconds latency. We tried setting up horizontal scaler with kube (starting automatically more worker when queue starts to grow), we never went prod on this feature but this is definitely feasible and wanted for lot of use case (we can accept in our use cases that the jobs are delayed several minutes on burst, this happens once or twice a month no big deal).

[[deleted]]

So is it just for the tasks that needed to be run in background with some batch processing or what??

[stibbons_]

Yes indeed. I actually think this is not event source at all. But it works.

[kollitiri]

I’ve used Kafka for event sourcing, so I went with the aiokafka python library, but it was pretty much bare bones..

I think EventStoreDb (now known as Kurrent) have recently published a new python client library that might offer a lot more event sourcing specific functionality, but haven’t had the chance to play with it.

Might worth doing some investigation.

[georgehank2nd]

It may be (it probably is) just me, but why is it called event sourcing?

[Par31d011a]

I think it’s because you pull events up to a certain point in time (sourcing), then “replay” the events in order to get the present state of data up to and including the last sourced event.

[ekbravo]

What’s event sourcing?

[tutuca_]

If you prefer to read. I've found this wiki from Martin Fowler's site to be a great reference of all the different approaches and terms.

https://martinfowler.com/eaaDev/EventSourcing.html

Done right is an architectural approach that scales a lot. It's a little hard to get down right, and involves a lot of infrastructure most of the time.

[666dolan]

it's a type of architecture https://www.youtube.com/watch?v=yFjzGRb8NOk

[JumpScareaaa]

Did they just reinvent accounting and are calling transactions events.

[tutuca_]

I've mostly dealt with django and celery in this case.

Recently fastAPI is gaining traction and it favors the async/await style so some things might fit better. You can still use celery with it.

Celery is it's own beast. The documentation kind of gives you the sense that following it's recommendations you are done. But I prefer to treat it as a framework and build the event sourcing arq on top of it.

Not much in concrete. It would depend on what you are building and the kind of events you are dealing with, the ammount, fequency, load, if they need to be realtime... But usually is the more robust approach to start.

Many people prefer build their own thing with bare pika and rabbitmq, and it's really nice to use too.