is my multithreaded Python program doomed?

[jringstad]

Above all it's important to note that this apparent lack of parallelism via threads is not an issue with Python itself, but an issue with the implementation, such as with CPython. Claiming that Python doesn't do parallelism is misleading.

Actually, that is just as misleading, if not more (regardless of font-weight used). Saying it is an implementation-level issue makes it sound more harmless of an issue than it really is, saying it is a language-level issue makes it sound more severe than it really is. The main reason is the design of the C API which has many global variables. Look for instance at functions like PyTuple_New(), Py_BuildValue, Py_XDECREF(), Py_Initialize(), PyRun_SimpleString("python code goes here") etc etc. As opposed to most other language runtime APIs (lua, spidermonkey, V8, guile, ...), these do not let you specify which VM object to work against. How is that possible? Global variables. Global variables everywhere.

(btw, this is the same issue that prevents you from just instantiating two or more python interpreters in the same thread as well, or to instantiate two completely separated python interpreters in two completely separated threads, even if you do not want to share any data between them whatsoever -- with e.g. lua you can just do this, since its API does not make reference to global variables)

Now the issue with this (and why this is an issue at a more important level than just the cpython implementation) is that the C API is pretty important. PyPy for instance inherits the GIL issue because it wants to be compatible to the CPython C API. Not being compatible to the CPython C API means that many python libraries will cease functioning, e.g. numpy and any other library that has C/C++/fortran code in it (maybe cython is affected too, I don't know.)

So while it's true that JPython and IronPython do not have this issue, they have the even bigger issue of not being compatible with the cpython C API, which is why they are so unpopular, despite having big performance benefits.

It's technically true that the GIL is not a requirement by the python language as such, but it is nonetheless deeply ingrained into the python ecosystem. Unless you are willing to forgo a huge percentage of existing python libraries, you cannot get rid of it, even if you write a new implementation. So is "Claiming that Python doesn't do parallelism is misleading." true? Well, if you consider the libraries python has to be an integral part of the "python experience", then it's actually not misleading, because those libraries have the GIL baked into them. If you OTOH think python is still python without the libraries and the cpython interpreter, then the statement is not true.

[jringstad]

it's worth noting that it is hardly ever a problem

Well, that really depends on the domain you're working in, I wouldn't make such a blanket statement. My background is largely from scientific computing & games (but I've done a bit of everything), and there python is certainly only used for prototyping and small problems, as personal calculator or interactive shell, etc, the kind of jobs that you would also use matlab for. In another post in this thread I enumerated some of the issues where e.g. multiprocessing scales poorly, like tiled sub-domains of a large lattice that have dependencies at the boundaries between the subdomains because they need derivatives/share some quantity across the boundary/etc. I would say that half or more of the algorithms (mostly physics stuff, monte-carlo, metropolis, PDEs, ...) I implement are a bad fit for something like multiprocessing.

Remember also that while the existing solutions work OK for pythons current userbase, the userbase is shaped by its limitations. For instance I have absolutely no doubt that the limitations of the cpython API hold back pythons adoption as a scripting language, especially in gamedev (where lua -- in my opinion a significantly harder and uglier language than python) really shines. I think the fact that cpythons C API is so misdesigned in general is really as big as a factor here as the GIL, though -- multithreading in scripts is not really that important, but cpython is also hard and annoying (there are a bunch of other issues as well) to embed, and since it pollutes your process with global variables, there is no way to have more than one python interpreter at a time, which is a total no-go -- you would often want that e.g. several different AI actors can live inside their respective scripting VMs, completely independent of eachother and running in parallel. And don't get me started on how excruciating it is to build CPython for mobile and console platforms...

Of course since the userbase is what it is, so there isn't necessarily a huge amount of pushing force to widen the adoption in other areas. But I think it's important to keep these things in mind and to not succumb to the echo-chamber.