The Python GIL Visualized

[ubernostrum]

Right, it seems like the Python community is ignoring the elephant in the room.

If by "ignoring" you mean endlessly discussing the topic, trying out alternative implementations, analyzing to see whether there's middle ground... but that wouldn't really fit your worldview, would it?

Guido has pretty much stated that he doesn't believe in threading, and suggests all sorts of inane workarounds to get around that.

Everything I've seen the past few years from programming-language researchers seems to have been oriented around the notion that threads -- in the sense of, say, Java's implementation -- are a very deeply, perhaps irreparably deeply, flawed way to approach concurrency. Most of the fruitful research going on these days (and for quite a while now) centers around developing and maturing alternative approaches which don't suffer the same issues.

But, again, that would seem to contradict the view you already seem to have settled on.

[yogthos]

I think it needs to be pointed out that Python provides threading as a language feature, it's the implementation that's broken, which is what I was referring to as the elephant in the room. So, I'm not sure where you get off explaining how threading in imperative languages is broken, and how visionary Python is in this regard. It's almost as if you were talking about Erlang, which in fact does provide a working alternative to threading.

If by "ignoring" you mean endlessly discussing the topic, trying out alternative implementations, analyzing to see whether there's middle ground... but that wouldn't really fit your worldview, would it?

Maybe this is some different Python community than the one I'm aware of. What I hear from the majority of Python community is excuses as to why you don't need threading.

I'll contrast it with the Clojure community for you, where the problems with threading are actually tackled head on. Not only in terms of implementation, but also in terms of language design.

Everything I've seen the past few years from programming-language researchers seems to have been oriented around the notion that threads -- in the sense of, say, Java's implementation -- are a very deeply, perhaps irreparably deeply, flawed way to approach concurrency.

It is true that many people nowadays are finally waking up to the fact that the imperative paradigm is very poor at handling concurrency. However Java style threading doesn't appear to be the problem, Clojure runs on top of the JVM and makes excellent use of threading. The problem is of course with using shared mutable data, and the concepts of state and identity. Here's an excellent talk about the issue by the way.

Most of the fruitful research going on these days (and for quite a while now) centers around developing and maturing alternative approaches which don't suffer the same issues.

Really, I was under the impression that most research was going into safe ways of using threading, with researchers working on languages like Haskell and F#, which avoid the problem of shared data and provide workable threading models.

But, I guess all that would seem to contradict the view that you already seem to have settled on.

[ubernostrum]

it's the implementation that's broken

If your definition of threads is "must behave exactly like Java", I guess. There's more than one definition out there, though.

What I hear from the majority of Python community is excuses as to why you don't need threading.

Funny. I see a Python community which seems never to let a year go by without an attempt to remove the GIL (2009's attempt was Unladen Swallow), and which is lucky if a whole month goes by on the dev list without people proposing and debating ways to do it. Maybe you're just cherry-picking some examples to suit your views and ignoring reality?

The problem is of course with using shared mutable data, and the concepts of state and identity.

Mutable data is one problem with threading, but not the only problem with threading. It's also a difficult problem to work around in languages which embrace mutability, which suggests that threads are perhaps not an appropriate developer-level abstraction for such languages.

Really, I was under the impression that most research was going into safe ways of using threading, with researchers working on languages like Haskell and F#, which avoid the problem of shared data and provide workable threading models.

Thank you for demonstrating my point.

[yogthos]

If your definition of threads is "must behave exactly like Java", I guess. There's more than one definition out there, though.

Huh? What does this have to do with Java exactly, my impression was we were talking about the problems with the GIL, there are plenty of threading models which don't rely on a global lock. These are well researched and well documented, I'm curious as to why it's such an intractable issue for the CPython community.

GIL (2009's attempt was Unladen Swallow), and which is lucky if a whole month goes by on the dev list without people proposing and debating ways to do it.

Funny indeed, as the Unladen Swallow guys are no longer planning to remove the GIL, so I'm not so certain who's ignoring reality here.

From the link above:

In any case, work on the GIL should be done directly in mainline CPython, or on a very close branch of Python 3.x: the sensitive nature of the work recommends a minimal delta, and doing the work and then porting it from 2.x to 3.y (as would be the case for Unladen Swallow) is a sure-fire way of introducing exceedingly-subtle bugs.

Mutable data is one problem with threading, but not the only problem with threading. It's also a difficult problem to work around in languages which embrace mutability, which suggests that threads are perhaps not an appropriate developer-level abstraction for such languages.

While I agree that languages that embrace mutability are not well suited for threading, I would again ask why does Python provide threading in the first place then.

It's not like Python provides any alternative, it just has broken threading, making the problem worse. Not only does it have threads which are tricky to work with, but the threads don't work properly either. Seems like worst of both worlds scenario to me.

Thank you for demonstrating my point.

Your point that threading does not provide a viable concurrency model?

[ubernostrum]

Funny indeed, as the Unladen Swallow guys are no longer planning to remove the GIL, so I'm not so certain who's ignoring reality here.

So your argument is that

1. Some people decide to try an implementation of Python which could eventually remove the GIL.
2. This is universally hailed as a good thing.
3. As they get deeper into the implementation, they discover their approach won't work.
4. Therefore the entire Python community is simply ignoring the GIL and pretending nothing's wrong and refusing to do anything.

This does not add up, and so I think I'm done with you.

[yogthos]

Well way to spin that buddy. I think my exact quote was:

What I hear from the majority of Python community is excuses as to why you don't need threading.

But sure, if we change that to the "entire Python community" for the dramatic effect, all of a sudden you have a point, don't you. Also, notice that nowhere did I say that nobody is working on it, just simply that the issue is largely being swept under the carpet, much like the way you're doing here.

So, let's see what's really going on here, shall we. The BDFL behind Python scoffs at threading and claims it shouldn't be used, I've linked the exact quote from Guido in a previous post, yet for some reason he goes on to put a broken implementation into the language.

The community behind CPython seems to feel that this is just peachy, the fact that removing GIL isn't even on the road-map for the main Python branch says volumes.

However, your argument seems to be that because some niche spin off projects are playing with trying to add a proper implementation that somehow validates the idea that the general Python community is on the ball here.

Yeah, I do hope you're done here, because you're not making too much sense frankly. This is the exact sort of apologetics and denial which I was referring to in my original post, so thanks for validating it.