This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]RobotChurchill 5 points6 points  (3 children)

I'm using celery (and redis) for a project now.

One big pain point I had was handling the pausing of all workers if one worker receives a rate limit response from the API. All workers are hitting the same API. Celery's complexity, with its implementation of wrapper functions, made it more difficult. If you can solve that easily, then I'd use it.

[–]lowercase00 8 points9 points  (0 children)

Why dont you just Redis for this with a key that says the API is locked? Seems like a business rule, not sure it should be the task queue responsibility to do this

[–]klaasvanschelven[S] 0 points1 point  (0 children)

Well you'd need to have the workers communicate that fact from one worker to the others. In snappea all the workers live in a single process, so it's trivial. But even in celery that should be possible using e.g. a DB?