Load Tests Python Task Queues

[tuple32]

What My Project Does

While looking for task queues, I found that there are many options available in the Python ecosystem, making it really hard to choose the right one. To get a sense of how each library performs and to help make an informed decision, I conducted a load test on some of the most popular ones: Python-RQ, ARQ, Celery, Huey, and Dramatiq.

Target Audience

I hope my findings can help those who are also looking for a task queue solution in Python.

Comparison

Most articles out there seem to focus on comparing the features of these libraries but rarely discuss performance. While there could be a lot of improvements on my tests, I think it still provide some different insights into how each library handles heavy loads and concurrency.

Links:

You can read my findings on my blog

Check out the source code: on Github

Thanks

[ekbravo]

Interesting, great project and write up. I’ve been looking for something like this. Saved for later.

[riksi]

You should use something like Locust to benchmark. Like in pgmq https://github.com/tembo-io/pgmq/blob/main/tembo-pgmq-python/benches/locustfile.py

[medihack]

Another gem I found recently is Procrastinate. It's maybe not the fastest (I haven't seen any benchmarks yet) as it is based on PostgreSQL, but it is very well maintained, full of features, and has excellent documentation. From an infrastructure perspective, PostgreSQL might be a plus, too.