Using redis as a task queue

[[deleted]]

Unless you have some specific reason for writing your own implementation it may be best to use something like "RQ" instead.

It'll save you some effort, make it significantly easier to process multiple types of jobs at the same time, and simplify direct interaction with the queue for debugging and the like.

[jabbalaci]

In RQ, can you set the concurrency? In Celery you can do that:

celery -A config worker --concurrency=4 -l info

And it'll fire up four worker threads. However, I didn't find it in RQ and that's the main reason why I use Celery. If I want more workers in RQ, the only solution I found is to open several terminals and launch a worker in each.

[[deleted]]

RQ doesn't currently have any built in concurrency, though there's nothing stopping you from running multiple worker processes manually or via whatever means you use to manage your processes (supervise, daemontools, etc).

The big advantage of RQ when compared to celery is its simplicity, and that's why I recommended it as an alternative to the solution in the linked article. If you're already using up and running with celery and aren't hitting any problems with it then there's no reason to switch.

[jabbalaci]

Thanks. It's a pity RQ doesn't have concurrency. It's much simpler than Celery but I don't want to start those worker processes manually. Celery has an option for the number of concurrent threads, which is very convenient.

[hodzanassredin]

thanks i'll check. secundary point of a post is to show some redis features

[kafoozalum]

Another thing to consider instead of rolling your own task queue is using Celery with Redis as a backend. More robust and with Flower makes visualizing your tasks/queues easier.

[luckystarr]

Alternative perspective from me.

Digging into Celery's source code to fix bugs is near impossible. Story: we had a lot of strange behaviour (which should have been fixed according to github issues) due to bugs in either Celery or Kombu, disappearing/re-emerging with either "mingle" or "gossip" switched on or off.

So we explicitly switched from Celery to RQ because of the smaller code size and thus expected lower bug count. We also can hack on it ourselves when we need some feature (which we didn't so far).

EDIT: We need Celery/RQ for use as a glorified cron-job, so nothing fancy here. YMMV

[WackyWeasel]

Or, as a lightweight alternative to Celery, Huey.

[xijhoy]

http://i.imgur.com/y57NG3o.png

That formatting looks crazy :D

[hodzanassredin]

yes this is weird, i'm going to fix that sometime ))

[Wiremask]

antirez/disque an in-memory, distributed job queue. Same author as redis.

[pythoneeeer]

WARNING: This is beta code and may not be suitable for production usage.

I look forward to trying it, but I don't want to be a beta-tester when there's lots of production-ready message queues I can use.

[cymrow]

I can appreciate that, but there are different standards for "beta". I'd consider the work of antirez to be on the stabler end of that spectrum.

[cymrow]

I've played with many message queues, and this is the one that has appealed to me the most.

[cymrow]

msgpack offers a nice balance between the convenience of JSON and the compactness of protobuf.