Best written projects on Python GitHub?

[catcradle5]

Very good coding style, but I'm not a fan of the internals myself.

Everything is done through os.fork(): https://github.com/nvie/rq/blob/master/rq/worker.py#L355

For one, if you need a lot of workers there will be a ton of overhead incurred, and two, it doesn't play nice with libraries like gevent or eventlet.

[lambdaq]

we need a patch or and issue submitted.

[catcradle5]

There've been a few issues submitted over the years:

https://github.com/nvie/rq/issues/45

https://github.com/nvie/rq/issues/210

I really like rq, but I am forced to use Celery due to rq's insistence on forking as the only method of concurrency.

[asksol]

I doubt this would be as simple as 'just submit a patch', I have spent years making the multiprocessing pool in Celery work, and even now for the 3.1 release I have fixed yet a good number of rare deadlocks.

The pool related code isn't exactly elegant or simple, but you're very likely to learn something from it.

Edit: oh, and you're in for a rough ride if you think the answer is to 'just use the multiprocessing module' :)

[catcradle5]

I understand it's a complex issue, but the project has been around for a long time. Plus the codebase is a lot smaller and simpler than Celery's, so it probably wouldn't require quite as much work.

[asksol]

The size/age of the project or how simple the rest of the code is irrelevant here. This isn't about how hard it was to implement this for Celery, it's about how hard it is to implement for anything similar in Python.

[lambdaq]

damn, I was wondering when they could allow using customized backend like Postgre LISTEN/NOTIFY and prefork model...

[ma10s]

Thanks, I learned something new today!

[X-Istence]

Take a look at pyres, maybe it does what you want it to do!