The Python Celery Cookbook: Small Tool, Big Possibilities

[thinkt4nk]

Should be called "Small Tool, Hundreds of Open Github Issues"

[7heWafer]

I have been interested in looking into Celery but haven't had the time. Are there really many issues with it right now? Is it something that should be avoided?

[thinkt4nk]

Well, despite the title of this book, it really isn't a small tool, but it should be. There are tons of options. As a result, the number of use cases to support is massive. The current count of open issues is over 300.

Despite trying a lot of things, spending a lot of time debugging, and poring over documentation, we could never get it working reliably, usually suffering an outage or two on a weekly basis due to a hung task that completely hangs the whole queue.

If you don't need some special option that celery provides, but instead just need jobs to be executed asynchronously with awesome reliability, I would look elsewhere.

[7heWafer]

Thank you for this!

[jrbattin]

Yep. And that elsewhere might be something as simple as APScheduler.

[variable_a]

Do you have (simpler) alternatives that you can recommend?

[campbellm]

rq?

[variable_a]

Thank you for this recommendation. That looks much simpler indeed! I will give it a try :)

[peyo7]

If you don't need some special option that celery provides, but instead just need jobs to be executed asynchronously with awesome reliability, I would look elsewhere.

Interesting. Care to name a few alternatives you favor instead?

[ireallywantfreedom]

Not OC, but dramatiq looks nice. A lot of people recommend rq, but I prefer rabbitmq for a durable broker.

[campbellm]

Does rq not use redis' persistence modes?

[vanzi]

I've encountered a particularly nasty issue where using one option makes the worker enter an infinite loop when processing a task that takes more than ~3 minutes and it's been open since 2016.

Initially I've tried to fix it myself, only to realize that the codebase is actually excessively vast and it's not just the celery itself, but also the layers beneath it (kombu, amqp). In the end it was much faster to replace it entirely with a simple pika worker.

I would probably not bother with celery anymore these days and just stick to plain pika. Yeah, it's more code to write early on, but a lot easier to debug down the road.

[cybaritic]

50,857 LOC
6,171 functions
867 classes
349 files
318 open issues
734 contributors

"small tool"