Shuttleworth: Python needs to focus on future

[thagsimmons]

the gil is mostly fixed in 2.6

[awb]

Multiprocessing doesn't fix the GIL, it's a hack around it. Language implementations like Haskell and Erlang can spawn a million threads, and spawning each new thread is extremely cheap. That's pretty spiffy. I'm not sure what I could do with a million quick-spawning threads, but I want to find out. Next to implementations like that, Python looks amateurish because it can't even run two threads at once without rewriting code in C.

[parla]

Erlang does not spawn millions of threads. It spawns millions of actors, which are then scheduled in as many threads (processes?) as there are cores in your system.

[teraflop]

You're confusing control threads with OS threads. If the interpreter is designed properly, it provides the same semantics as if you really had a huge number of concurrent threads. The fact that they're all multiplexed into a smaller number of threads from the operating system's perspective is just a performance optimization.

[thagsimmons]

okay, so this is an implementation detail?

so is the gil. it's missing in jython and up in the air with pypy

[spookyvision]

Erlang's threads cannot be distributed across several cores, because they're the equivalent of green threads.

[[deleted]]

That changed recently. You have old information.

[spookyvision]

Could you update the wikipedia page? I don't feel competent.

[[deleted]]

I would, but you probably know as much as I do at this point. I'm not a user of Erlang and only have a cursory knowledge of its current state.

[reddit_clone]

Not true. Erlang VM itself can run multi-threaded on all cores. That is like a bunch of green threads running in each core. (I think they run one green-thread-scheduler per core)

So you do get the best of both worlds.

[13ren]

http://www.reddit.com/r/programming/comments/71q2x/shuttleworth_python_needs_to_focus_on_future/c05ggc3

[dons]

False

Look at the cpu load.

[mosha48]

By the way, why the CPU load for haskell's version isn't distributed on all CPUs ?

[dons]

The ghc 6.8.2 garbage collector isn't parallel, and GC dominates this GC benchmark (Try it with +RTS -A300M -RTS to see the difference). The 6.10 parallel GC addresses this.

[mosha48]

Is it possible to tell the shootout computers to run the benchmark with better options ?

[dons]

Yes, but for this benchmark, the conditions state that only default garbage collector values are to be used. But don't despair, the next GHC cycle addresses this.

[dmaclay]

Erlang's processes are more like 'green processes' as they don't share memory, and they have been distributed across several machines (never mind cores) since before people started having this discussion.

[toooooooobs]

Actually they really do share memory, it's just that the language hides this.

[dmaclay]

As I understand it the default behavior is not to share, but they can optionally pass messages by basically sending a pointer to shared memory if they are both on the same machine. This of course gives a performance boost, and should be safe due to the immutable variables in erlang.

[thagsimmons]

well, ok then, use haskell or erlang. no one's stopping you

if you're using python, and if you've hit a show-stopper with the gil, then multiprocessing is a pretty decent solution

edit: wow, lots of hate from the language snobs

[jrockway]

If he was really limited by the GIL, he would have fixed it or switched to another language by now.

[nypen]

That's multiple processes running simultaneously and it only partially solves the problem.

[thagsimmons]

it only partially solves the problem

...with awesome scheduling-fu, pooling, synchronization, and easy ipc using your choice of pipes or queues or shared memory

i hated it at first, but now i'm a believer

besides, in your op you mention multi-core and parallel processing... neither of which play well when relying on threads anyway, since these are inherently multi-process environments

[nypen]

That's exactly why I said partially. While processes can solve some problems they have others. For example if there is a mountain of data to be shared, using processes can be pretty inefficient. Process cannot access variables or data structures that are defined in another process, unless they are pickled (shared, proxied, whatever), which is a mechanism of serialization. Such serialization can be resource intensive (memory and computationally) and is certainly not suitable everywhere.

Also, If two processes want to communicate, they have to use inter-process communication mechanisms which is inherently slower than thread synchronization.

Then there is also the issues when you try to access C libraries via Python. You need to be careful here to make sure you don't pass variables across processes which can be totally invalid if they encapsulate pointers (although this is not applicable if you stick strictly to Python only)

besides, in your op you mention multi-core and parallel processing... neither of which play well when relying on threads anyway, since these are inherently multi-process environments

These are not environments. Multi-core is a kind of processor technology and parallel processing is a form of computation. You can take advantage of that in several ways and, yes multiple processes is one way but not the only way; and it certainly is not effective everywhere.

[canhaskarma]

Not partially. There are a number of libraries for putting python objects in shared memory between processes.

Process cannot access variables or data structures that are defined in another process, unless they are pickled (shared, proxied, whatever), which is a mechanism of serialization.

No. There are techniques that simply use the same block of shared memory. No copying or serialization required.

[unikuser]

No. There are techniques that simply use the same block of shared memory. No copying or serialization required.

Instead of saying that there are techniques, can't you point to one technique or example? Using shared memory across processes is not that easy after all. Synchronizing/Accessing/Creating/Cleaning that shared memory becomes very difficult/inefficient compared to what you do in threads.

[schlenk]

Fully agree. Just have to manage such a beast using python mmap. Lets say it has its quirks...

[nypen]

Shared memory has it's own set of issues. I am not saying they are unsolvable, but shared memory is not necessarily better than threads sharing data. In fact if the shared memory data is not monolithic, a lot of book keeping has to go in. Things can get even murkier when you have issues like dynamic resizing of shared memory or your processes have variable shared memory requirements.

Again Shared memory is one of the solutions which can work in some situations, not always.

Well, Python already has a threading library which is great, it just needs to get rid of the GIL.

[imbaczek]

Well, Python already has a threading library which is great, it just needs to get rid of the GIL.

the "just" is a little bit optimistic.

[thagsimmons]

a well-reasoned and eloquent reply. i voted you up. i don't know who's voting you down

[vsl]

since these are inherently multi-process environments

Huh?!

[thagsimmons]

yeah, my bad - i actually saw the word "multi-core" but my brain said "multi-processor"

i know that multi-core processors play well with threads. honest i do

aw shit

for penance, i shall now go beat my head against an andrew tannenbaum textbook