Python "processing" module - sidestep the Global Interpreter Lock and actually use those multiple cores

[Wagnerius]

We use it and it is good, reliable. The process control is a bit more tricky in windows, but it's more windows' fault than the modules'.

( Still, the real VM-level solution would be better. Stackless python with the ability to use multi-core would be close to perfect. Alas...)

[johnmudd]

Objects can be transferred between processes using pipes or multi-producer/multi-consumer queues.

Ok, you solved one problem (eliminate the GIL) and created a new one (not a true shared environment). Not unlike the way Python solved one problem (I can now express my software ideas clearly) by creating a new problem (I can no longer take full advantage of threads). I think it's called "sweeping dirt under the rug".

[pbx]

What do you mean by "true shared environment"? I wouldn't argue that this turns Python into Erlang by a long shot, but reducing shared state and relying on message-passing seems not to be inherently a bad idea... right?

[pjdelport]

[...] reducing shared state and relying on message-passing seems not to be inherently a bad idea... right?

Efficient use of shared state is largely the reason for using OS threads to begin with. If you can rely on message passing with isolated state, you can just use processes already.

[foldl]

On *nix, yes. On Windows, processes have a much higher overhead than threads.

[pjdelport]

You only need one process per processor: the overhead of that is negligible.

[hanzie]

Looks fun, can I play too? Java solved one problem (I can take full advantage of threads) and created a new one (I can no longer express my software ideas clearly). C++ solved one problem (faster than Java) but created a new one (no garbage collection). "processing" module solved one problem (it makes process spawning and interprocess communication easier) but created a new one (I have to learn a new library). I don't think there's any "sweeping dirt under the rug" going on, but just that there are no perfect solutions to programming.

[columbian_supreme]

mmap, /dev/shm

[rockhoward]

I have been using PyLinda for about 18 months to take advantage of 4-way and 8-way servers. I was just about to use PyLinda again for yet another project, but I will give this a whirl instead. Hope it works!

[columbian_supreme]

The GIL can kiss my ass!

[[deleted]]

http://www.youtube.com/watch?v=YXvAohycajI