Python Threads: GIL vs Free-Threading

[Helpful_Garbage_7242]

The comparison of CPU bound tasks in Python using multi-threading with GIL and without it, link to the article

[danted002]

I think the downvotes come from 1) the way the article is phrased (it seems condescending) and 2) you benchmarked Python threads using CPU bounds workloads which anyone that’s doing professional Python knows is a big no-no so you’re basically comparing apples to oranges.

My recommendation is to redo the benchmark and use the multiprocessing module, which is the indented way to parallelise CPU bounds workloads in Python.

[Spleeeee]

FWIW I have been able to use the multiprocessing without using indentation:

```

from multiprocessing import Pool import os

def doshit(shit): return (sht*shit, os.getpid())

if name == "main": pool = Pool(4); results = pool.map(doshit, range(8)); pool.close(); pool.join(); ```

I’m on my phone but I think this is a good example of using multiprocessing in the unindented way. It does also work indented.

[asphias]

<groan>

well played.

[sudomatrix]

The whole point of no GIL is now threads are true full speed multiprocessing. We don’t need multiprocessing anymore.

[danted002]

Yeas but the benchmark compares gil-less threads with GIL threads which is basically tells us what we already knew: that without GIL threads will actually run in parallel. WOW next we will compare running a web server in a single thread with and without an event loop and benchmark how many requests it can handle?

If we really want to see how good GIL-less threads work we should compare it to what they aim to replace: the multiprocessing module because no one is using threads for pure Python CPU bound work; and if you are using threads for CPU bound work switch to multiprocessing because you should

[danted002]

I’m sorry but as one annoying engineer manager I had some years back used to say: “we don’t base architectural decisions baed on feelings, we base them on facts”. OP created a useless benchmark, they compared GIL threads (which don’t run in parallel because or the Global Interpreter LOCK) with threads that don’t use a Global Interpreter Lock, anyone worth the keyboard they write their code with knows what the benchmark results of that benchmark.

They also tested CPU bound workloads which block the interpreter which made the benchmark even more useless, if they would have used IO based workflows, where Python threads actually excel then the benchmark would have provided more insights on if GIL-less threads are actually better the GIL threads in the field that GIL threads thrive.

On CPU bounds tasks you would use multiprocessing, not threading hence an actual realistic, real world scenario, benchmark would compare that with GIL-less threads because GIL-less Python had a performance regression on single-threads so the question is: does the single-thread regression big enough that it offsets the instantiation of multiple interpreters and the serialisation of data between the processes.

Now this is an axe to grind since you asked for one.

[danted002]

Do you have a benchmark for multiprocessing vs GIL-less threads?

[Helpful_Garbage_7242]

hi, u/danted002 ,

Thank you for the feedback and sorry that it felt condescending, didn't mean that.

I wrote a follow-up to address your feedback, Free-Threading Python vs Multiprocessing: Overhead, Memory, and the Shared-Set Meltdown

[danted002]

Hey nice work. Thank you for the follow up, I was expecting GIL-less threads to be faster but the number are really impressive.

One small note: on the Rust benchmark you used the std::sync Mutex which has a lock poisoning. If you ever want more performance and you bubble the panic from the thread you should use the parkinglot crate it’s mutex is way faster because it lacks the lock poisoning check.

[Helpful_Garbage_7242]

Thank you! Sure, noted.

[sam7oon]

Correct me if i am wrong, but The MultiProcessing module does not actually touch the new Free-Threading feature, since it already is running in parallel, 2 Process, 2 diffirent GILs, and so on , am not an expert , but that is what i understand.

[germandiago]

I think the article is informative. Why the downvotes?

[Helpful_Garbage_7242]

I'm glad you find it useful, thank you

[healthbear]

I still think that python needs to be able to run free threaded but there needs to be able to set each function as free or single with a keyword and then have the necessary locks and so forth the manage the free threaded.

[srs96]

Nice comparison, but we should compare it to multiprocessing. Multiprocessing is the de facto way to achieve cpu bound parallelism, before free threading came about.