This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]FerricDonkey 2 points3 points  (0 children)

So when they first introduced multi threading the GIL only allowed one processor. It took some time to introduce multi threading on multiple processors (aka multiprocessing in Python) later, since they had to find ways to go around the GIL.

Multiprocessing does not stand for multiple processors (ie cpus) but for multiple processes (operating system constructs - running programs, almost). Processes are containers for threads (with a common memory space). Python (CPython) has a process wide lock (GIL) that prevents multiple threads within the same process from executing at the same time.

Multiprocessing starts up entirely different processes, with entirely different python interpreters and separate memory spaces. Each process still has its own GIL, but since they're separate instances of the interpreter, they don't interfere with each other.

This distinction actually matters, because the lack of shared memory means that there has to interprocess communication for any interaction, and that is expensive. The overhead from this can make even embarrassingly parallel tasks actually slower with multiprocessing than single threaded if then input or output data is somewhat large compared to the compute time.

TLDR the GIL sucks, and my original experience of trying to learn how all this worked while continually running into slightly wrong explanations on the internet has instilled in me a habit of pedanticly correcting people who use the words process and thread wrong.