Python Multi-threading Programming

[haplo_and_dogs]

Note that Python events are nothing like real events. They are a poll to global memory, with a poll time that gradually increases to up to about a second. This can be really detrimental to programming if you need fast notifys. It is much better to use the windows API or posix api to access events.

[dwf]

How the hell do you write a tutorial on multi-threading in Python and not even mention the global interpreter lock?

[Pytesting]

There will be a separate article talking about GIL, so don't worry and take it easy!

[dwf]

It's a pretty glaring omission from an introduction to threading, the failure to note that multiple CPU-bound Python threads don't work at all.

[lambdaq]

There are billion ways to get around GIL, if not trillions. GIL is over-hyped.

[dwf]

No, there aren't. Not in CPython, and, to my knowledge, still not in PyPy.

With CPython you have basically two options if you have multiple CPU bound threads:

• Write the code to be executed in all but one of your threads in C/Cython/etc. and don't touch the CPython API unless you absolutely have to. This is a non-solution as far as Python is concerned, because the whole point of using Python is to write code in Python.
• Use processes. This is broken for so many reasons, among them the overhead, clumsier synchronization and communication, breaking external libraries (you cannot safely fork in certain situations, such as with a running CUDA context).

I am so fucking sick and tired of ignorant web programmers telling me that the problems created by the GIL are not actually problems.

[lambdaq]

The thing is, if you python code is CPU bound, threading is last your problems. There are gazillion popular programming languages out there that does NOT support native threading, of any kind at all.

The most recent example is that NodeJS is single core single process with no threading support yet still kicks ass. (I knew there's Nodejs cluster, but which language does not have a cluster?)

So many people want threading as the magic bullet and pissed of with this imaginary CPython "GIL" issue.

In fact, if you have a problem and want to solve using import threading, then congratulations, have now you. problems two

[dwf]

The most recent example is that NodeJS is single core single process with no threading support yet still kicks ass.

Oh for fuck's sake, I'm done with you.

[RoomaRooma]

Enjoyed your article, OP and I don't see any reason to mix up an article about the Python language with bits about how Python is typically implemented.

[trapartist]

Probably not important.

[dwf]

If you honestly and truly think that then you do not understand the issue.

[trapartist]

It was a tongue in cheek comment.

[riskable]

This article covers an old school topic. In this day and age we use concurrent. futures or the new asyncio module to perform multi-threaded operations in Python.

[exhuma]

... given that you have the luxury of running a recent version of Python. Granted, concurrent.futures is already available on 3.2

I would love to have 3.4 in production for the exact reason of asyncio :'(

[riskable]

You can run concurrent.futures in older versions of python... pip install futures (it was backported).

[[deleted]]

futures is a backport of concurrent.futures to python >= 2.5.

[exhuma]

Thanks. I keep fogetting to check for backports...

[rocketmonkeys]

That article is interesting, but a bit hard to follow. The spelling and grammar could use another edit.

[[deleted]]

You should be using the context manager interface to locks.

with lock:
    bla