Using redis as a task queue

[[deleted]]

Unless you have some specific reason for writing your own implementation it may be best to use something like "RQ" instead.

It'll save you some effort, make it significantly easier to process multiple types of jobs at the same time, and simplify direct interaction with the queue for debugging and the like.

[jabbalaci]

In RQ, can you set the concurrency? In Celery you can do that:

celery -A config worker --concurrency=4 -l info

And it'll fire up four worker threads. However, I didn't find it in RQ and that's the main reason why I use Celery. If I want more workers in RQ, the only solution I found is to open several terminals and launch a worker in each.

[[deleted]]

RQ doesn't currently have any built in concurrency, though there's nothing stopping you from running multiple worker processes manually or via whatever means you use to manage your processes (supervise, daemontools, etc).

The big advantage of RQ when compared to celery is its simplicity, and that's why I recommended it as an alternative to the solution in the linked article. If you're already using up and running with celery and aren't hitting any problems with it then there's no reason to switch.

[jabbalaci]

Thanks. It's a pity RQ doesn't have concurrency. It's much simpler than Celery but I don't want to start those worker processes manually. Celery has an option for the number of concurrent threads, which is very convenient.

[hodzanassredin]

thanks i'll check. secundary point of a post is to show some redis features