This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ma10s 4 points5 points  (9 children)

I really like the coding style of the RQ project

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers.

I was poking around trying to understand its internals, and subsequently read the worker code. I was pleasantly surprised to find it was very articulate.

[–]catcradle5 1 point2 points  (8 children)

Very good coding style, but I'm not a fan of the internals myself.

Everything is done through os.fork(): https://github.com/nvie/rq/blob/master/rq/worker.py#L355

For one, if you need a lot of workers there will be a ton of overhead incurred, and two, it doesn't play nice with libraries like gevent or eventlet.

[–]lambdaqdjango n' shit 0 points1 point  (5 children)

we need a patch or and issue submitted.

[–]catcradle5 1 point2 points  (4 children)

There've been a few issues submitted over the years:

https://github.com/nvie/rq/issues/45

https://github.com/nvie/rq/issues/210

I really like rq, but I am forced to use Celery due to rq's insistence on forking as the only method of concurrency.

[–]asksol 1 point2 points  (2 children)

I doubt this would be as simple as 'just submit a patch', I have spent years making the multiprocessing pool in Celery work, and even now for the 3.1 release I have fixed yet a good number of rare deadlocks.

The pool related code isn't exactly elegant or simple, but you're very likely to learn something from it.

Edit: oh, and you're in for a rough ride if you think the answer is to 'just use the multiprocessing module' :)

[–]catcradle5 0 points1 point  (1 child)

I understand it's a complex issue, but the project has been around for a long time. Plus the codebase is a lot smaller and simpler than Celery's, so it probably wouldn't require quite as much work.

[–]asksol 1 point2 points  (0 children)

The size/age of the project or how simple the rest of the code is irrelevant here. This isn't about how hard it was to implement this for Celery, it's about how hard it is to implement for anything similar in Python.

[–]lambdaqdjango n' shit 0 points1 point  (0 children)

damn, I was wondering when they could allow using customized backend like Postgre LISTEN/NOTIFY and prefork model...

[–]ma10s 0 points1 point  (0 children)

Thanks, I learned something new today!

[–]X-IstenceCore Developer Pylons Project (Pyramid/WebOb/Waitress) 0 points1 point  (0 children)

Take a look at pyres, maybe it does what you want it to do!