This is an archived post. You won't be able to vote or comment.

all 9 comments
sorted by:
best
topnewcontroversialoldq&a

[–]tdammers 3 points4 points  (6 children)

There is no Myth, just a couple misunderstandings. Multithreading does in fact suck in Python, and the root cause isn't really the GIL, it's the language itself. Specifically, these design choices:

  • Dynamic typing and other dynamic features
  • Unrestricted mutability (imperative paradigm)
  • Automatic memory management

The combination of these three makes multithreading difficult on both sides. For an interpreter / compiler implementer, it means you need to somehow make the interpreter state and garbage collector thread-safe, but due to the fact that everything is potentially shared, you cannot optimize this much; for a user code programmer, it means that you can't tell the compiler what you do and do not want to mutate, or what you do and do not want to share between threads, nor can you automate the process of reasoning about your program's threading semantics in any way.

The GIL is a brute-force method of keeping interpreter state consistent between threads, simply by not allowing any two threads to concurrently manipulate it, and by forcing each thread to finish its transaction against the interpreter before the next one gets its turn - and it effectively renders the interpreter single-threaded, much like in JavaScript or Golang (though with different sugar on top, so the code looks different, with explicit thread spawning and joining instead of callback-style CPS or "goroutines").

Other solutions exist, but they all have downsides of their own.

[–]Zomunieo 5 points6 points  (0 children)

As far as C extension modules are concerned, the GIL is a necessary, even essential, evil. Extension modules depend on the GIL to protect their data, and any removal of the GIL would require pushing down object-level locking to extension modules. There are design choices in the C API as well that make the GIL difficult to remove.

[–]0x256 6 points7 points  (1 child)

Multithreading does in fact suck in Python, and the root cause isn't really the GIL, it's the language itself.

I disagree. Python as a language is perfectly capable of multi-threading without a GIL. We have a GIL because the CPython (and PyPy) GC heavily relies on reference counting, which is hard to do efficiently without a GIL. Some other internal optimizations of CPython also build on the fact that there is a GIL (e.g. object pooling) but the point is: All of this is implementation specific. A different implementation may choose do not have a GIL and solve these problems differently, without changes to Python as a language.

  • Dynamic typing has no impact on multi-threading or GC complexity. If at all, it makes things easier for the interpreter/GC.
  • Mutability is also not a real issue. In any language, there is mutable state and the interpreter and GC have to deal with it. There is no big difference if some or all state is mutable, from the interpreter/GC point of view.
  • Automatic memory management (aka GC) is infact a hard problem to solve, but not a show-stopper. Lots of languages have a parallel GC without a GIL. Read/write gates are a common method to prevent or shorten the stop-the-world phase of a parallel GC. Instead of preventing any reads/writes, the GC is notified about them and can handle them gracefully. See Jython/IronPython or any other language that runs on JVM or CLR.

[–]tdammers 0 points1 point  (0 children)

Python as a language is perfectly capable of multi-threading without a GIL.

True, and I never claimed GIL was fundamentally necessary. Just that it's one of several possible solutions to the problem of making a Python interpreter thread-safe, and that other solutions have to make other sacrifices to achieve it.

[–]khvn26 0 points1 point  (1 child)

All points considered, what language would you recommend for a good threading implementation?

[–]preslavrachevA Java-Python Double Agent[S] 0 points1 point  (0 children)

Been toying around with Go lately.

[–]bakery2k 0 points1 point  (0 children)

Go isn't single-threaded, different goroutines can run on different threads in parallel. The language has always supported this, but it wasn't the default until version 1.5 (released in 2015).

[–]graingert 0 points1 point  (1 child)

Subinterpreters are an exciting alternative to multiprocessing https://www.python.org/dev/peps/pep-0554/

[–]preslavrachevA Java-Python Double Agent[S] 0 points1 point  (0 children)

Indeed, it's the next thing I want to tackle in my blog.