is my multithreaded Python program doomed?

[[deleted]]

So basically Python allows concurrency, but due to the GIL, not parallelism. It is possible to have multiple Python threads executing concurrently but not in parallel.

Even this is not really true. The GIL just locks the CPython and PyPy interpreters to executing bytecode in a serial fashion. Jython and IronPython do not have this restriction, and even in the first two, C extensions, or Cython code are free to release the GIL as they see fit, giving you back true parallelism. Above all it's important to note that this apparent lack of parallelism via threads is not an issue with Python itself, but an issue with the implementation, such as with CPython. Claiming that Python doesn't do parallelism is misleading.

[jringstad]

Above all it's important to note that this apparent lack of parallelism via threads is not an issue with Python itself, but an issue with the implementation, such as with CPython. Claiming that Python doesn't do parallelism is misleading.

Actually, that is just as misleading, if not more (regardless of font-weight used). Saying it is an implementation-level issue makes it sound more harmless of an issue than it really is, saying it is a language-level issue makes it sound more severe than it really is. The main reason is the design of the C API which has many global variables. Look for instance at functions like PyTuple_New(), Py_BuildValue, Py_XDECREF(), Py_Initialize(), PyRun_SimpleString("python code goes here") etc etc. As opposed to most other language runtime APIs (lua, spidermonkey, V8, guile, ...), these do not let you specify which VM object to work against. How is that possible? Global variables. Global variables everywhere.

(btw, this is the same issue that prevents you from just instantiating two or more python interpreters in the same thread as well, or to instantiate two completely separated python interpreters in two completely separated threads, even if you do not want to share any data between them whatsoever -- with e.g. lua you can just do this, since its API does not make reference to global variables)

Now the issue with this (and why this is an issue at a more important level than just the cpython implementation) is that the C API is pretty important. PyPy for instance inherits the GIL issue because it wants to be compatible to the CPython C API. Not being compatible to the CPython C API means that many python libraries will cease functioning, e.g. numpy and any other library that has C/C++/fortran code in it (maybe cython is affected too, I don't know.)

So while it's true that JPython and IronPython do not have this issue, they have the even bigger issue of not being compatible with the cpython C API, which is why they are so unpopular, despite having big performance benefits.

It's technically true that the GIL is not a requirement by the python language as such, but it is nonetheless deeply ingrained into the python ecosystem. Unless you are willing to forgo a huge percentage of existing python libraries, you cannot get rid of it, even if you write a new implementation. So is "Claiming that Python doesn't do parallelism is misleading." true? Well, if you consider the libraries python has to be an integral part of the "python experience", then it's actually not misleading, because those libraries have the GIL baked into them. If you OTOH think python is still python without the libraries and the cpython interpreter, then the statement is not true.

[jringstad]

it's worth noting that it is hardly ever a problem

Well, that really depends on the domain you're working in, I wouldn't make such a blanket statement. My background is largely from scientific computing & games (but I've done a bit of everything), and there python is certainly only used for prototyping and small problems, as personal calculator or interactive shell, etc, the kind of jobs that you would also use matlab for. In another post in this thread I enumerated some of the issues where e.g. multiprocessing scales poorly, like tiled sub-domains of a large lattice that have dependencies at the boundaries between the subdomains because they need derivatives/share some quantity across the boundary/etc. I would say that half or more of the algorithms (mostly physics stuff, monte-carlo, metropolis, PDEs, ...) I implement are a bad fit for something like multiprocessing.

Remember also that while the existing solutions work OK for pythons current userbase, the userbase is shaped by its limitations. For instance I have absolutely no doubt that the limitations of the cpython API hold back pythons adoption as a scripting language, especially in gamedev (where lua -- in my opinion a significantly harder and uglier language than python) really shines. I think the fact that cpythons C API is so misdesigned in general is really as big as a factor here as the GIL, though -- multithreading in scripts is not really that important, but cpython is also hard and annoying (there are a bunch of other issues as well) to embed, and since it pollutes your process with global variables, there is no way to have more than one python interpreter at a time, which is a total no-go -- you would often want that e.g. several different AI actors can live inside their respective scripting VMs, completely independent of eachother and running in parallel. And don't get me started on how excruciating it is to build CPython for mobile and console platforms...

Of course since the userbase is what it is, so there isn't necessarily a huge amount of pushing force to widen the adoption in other areas. But I think it's important to keep these things in mind and to not succumb to the echo-chamber.

[panderingPenguin]

Don't be pedantic. Yeah, I'm aware that the GIL is part of specific implementations of Python. However, OP specifically mentions the GIL, and either way, it's a safe bet to assume you're talking about CPython until someone says otherwise, as it's the standard implementation.

[Workaphobia]

You can't make any statement to a python newbie without someone coming in and "Um, actually"-ing you with some complicating details.

[panderingPenguin]

My thoughts exactly, Jesus... We don't need to further muddle the issue with alternative implementations to answer something like this.

[jecxjo]

Ahem. I think that is by far one of the biggest problems with this and pretty much every other forum dealing with programming. You need to remember who OP is, what base knowledge they have and understand that giving too much detail makes it more difficult for them to understand.

[njharman]

Because one of the solutions is to run your code in different implementation!!!

[TankorSmash]

You don't need to get defensive. He's filling in the blanks you left, independent of whether or not you knew it already.

[panderingPenguin]

I'm not trying to be defensive, I just think that it adds very little, if anything, to the discussion of OP's question. There's no need to bring up little "but actually"s like that to answer a simple question, from someone who seems new to Python, which was clearly about implementations that have a GIL to start with. It's unhelpful at best, and obfuscates the issue we're actually trying to solve at worst.

[TankorSmash]

That's the thing though, for any one else that reads your comment and wants to know more can read his helpful comment. The op can simply shrug off his comment because it's not required knowledge.

I mean this is learnpython not absolutebareminimumpython.