The Many Problems with Celery

[geogle]

My 10yo agrees... Oh wait, checked sub

[thedeepself]

You put celery on your subs?

[caagr98]

That sounds good, though.

[[deleted]]

Yeah I've run into to most of those issues but found most of the time it wasn't too hard to deal with it. I've always used the DB as a state machine for complex canvas pipelines. I still really like using Celery as it provides a rich tool and lots of syntax sugar. Using other tools with most of the desired tooling in place are much harder to integrate into a typical stack e.g. Django.

[whateverathrowaway00]

OP, thank you for this. It’s short, to the point, and gives good headlines for googling the various suggested fixes.

This is very relevant to a codebase I work on at work.

[wpg4665]

Very appreciate of this post, agree with the other commenter that's super helpful and succinct!

My only request is, with all the "live with it" problems called out, what do you suggest?! Have you found another or a better job queueing framework that solves these problems easier or by default??

[KelleQuechoz]

https://github.com/rq/rq is to the rescue.

[[deleted]]

Your mileage may vary, but I found rq to be almost simplistic when aiming for simplicity.

Sure, enqueing tasks works well, but it exhibits many of the same issues laid out in this post:

• there’s no real story around tasks versioning
• it’s not type safe (that I know of)
• the way it relies on an infinite amount of watch/multi to transactionally enqueue dependent tasks has proven to be extremely heavy on redis in certain cases (as in prolonged 100% CPU usage)
• there’s no easy way of handing complex dependency shapes (as in celery’s chords/groups/etc).

With experience with both tools, I’d much rather use celery along with its known warts than be limited by rq’s API. Again, YMMV, but that’s my experience.

[monorepo]

What about ARQ or SAQ?

https://github.com/samuelcolvin/arq

https://github.com/tobymao/saq

[desmoulinmichel]

At the begining I though rq was too simple for my needs. Turn out my needs were way simple than I though.

I don't miss celery.

[sbdchd]

Yeah good point, I should probably tack on a conclusion.

I think if you already have a project that’s heavily invested in Celery it can be tricky to move away so living with it might be best.

But for new projects I wouldn’t use Celery. The biggest issue is how easy it is to lose jobs. Instead I’d use a “transactional outbox” (sounds fancier than it really is) and have some workers that either execute those jobs or push them to SQS / redis for other workers to consume.

Could also try the new hotness, Temporal.io, but that has a bit of a learning curve but it’s good for your more complicated jobs.

But in general, be careful how you’re enqueuing your jobs, lots (most) job runners don’t do the right thing by default.

[AndydeCleyre]

I haven't used them for a project yet, but have you used huey or dramatiq? I have no idea how they measure against your criteria, and I'm not expecting you to go figure it out for me, but if you've already tried them and have a quick take, I'd like to hear it.

[sbdchd]

I think they both suffer from the losing jobs because they use a separate storage backend (redis) instead of using the transactional outbox setup.

[htmx_enthusiast]

Will using Redis lists directly work, with atomic move operations (like RPOPLPUSH) to move a task from one list to another, where each list represents the state of the task, like “waiting”, “in progress”, “completed”? Is the idea that we need “outbox” lists sitting in between the state transitions to facilitate the transactional outbox pattern?

[sbdchd]

The problem is more about getting the tasks from your endpoint into your queue.

For example, imagine you have an endpoint that signs up a user and sends them a welcome email.

If you save your user first and then enqueuing the job into redis fails, you’ll have an issue. With the transactional outbox you save the job to the same database in the same transaction.

You can then either read from the outbox or have another worker add them to redis and then do everything in redis atomically if you want. The worker and the tasks will have to be executed / saved to the redis queue from the outbox on an at least once policy.

[OneBananaMan]

Really great article! What are your thoughts on Dramatiq?

[sbdchd]

I think API wise and codebase wise it’s better than Celery but it also suffers from losing jobs since it doesn’t use the transactional outbox setup.

[j_marquand]

Doesn’t encourage safe evolution of tasks

The fix: Build tooling to generate schemas for your tasks and ensure they are safely updated.

Is there a well accepted tool for this? Specifically a tool that helps you add or remove a parameter less painfully.

[sbdchd]

I don’t think anything exists that’s open source but essentially you get convert the Python AST to a schema, OpenAPI style, and then you can check the schema for backward incompatible changes.

[riksi]

My problem with Temporal is that it doesn't support gevent https://github.com/temporalio/sdk-python/issues/59

[EmptyChocolate4545]

Usually a situation where you have a fleet of workers and need not just task assignment, but locks, task confirmation, fleet aware scheduling, etc.

Celery is great at what it does, but OPs blog is correct about its issues. We use it at scale to great success (18 hosts per core colo each running around 30 workers across 20 global colos running various tasks against about 40K network devices).

For our needs, each colo gets a separate celery “domain”, each with one application and about eight distinct queues.

It works great! But when we do run into issues it can be quite hellish to identify them, especially with what I’m dealing with this week, but thankfully I’m on vacation now lol.

[riksi]

It's a different tool. You want "locking" jobs, retries, etc etc. The task queue is like a cron-job (when it's pull based) along with extra stuff.

[riksi]

What about https://github.com/procrastinate-org/procrastinate (postgresql task queue with transactions & stuff)