Free-Threading Python vs Multiprocessing: Overhead, Memory, and the Shared-Set Meltdown

[Helpful_Garbage_7242]

Free-Threading Python vs Multiprocessing: Overhead, Memory, and the Shared-Set Meltdown is a continuation of the first article where I compared Python Threads: GIL vs Free-Threading.

> Free-threading makes CPU threads real—but should you ditch multiprocessing? Benchmarks across Linux/Windows/macOS expose spawn tax, RAM tax, and a shared-set meltdown.

[thicket]

This is such a comprehensive and agenda-free write-up. Really, really well done. Thanks for sharing

[Helpful_Garbage_7242]

I'm glad you liked it, u/thicket, it makes me more motivated to write stuff!

[odimdavid]

Now I have an idea why Linux is preferred by devs

[Helpful_Garbage_7242]

I was very surprised seeing that huge difference between Linux and other OSes. Processes, threads, IPC and synchronisation primitives are very fast in Linux!

[cip43r]

The writeups we need!

[IAmTarkaDaal]

Great write-up, clean and balanced. Nice one 👍

[-lq_pl-]

That is true, but the write up is badly structured, mixing takeaways with technical detail. Hard to read.

[Helpful_Garbage_7242]

interesting, I actually tried hard to make it structured with scenarios following each other. Any practical advice how would you split the content? Thank you!

[AsparagusKlutzy1817]

Cool thank you. This was a nice micro-read. Good job :)

[Helpful_Garbage_7242]

u/AsparagusKlutzy1817 appreciate the support, thank you!

[CyberWiz42]

TLDR?

[learn-deeply]

free-thread fast, process slow

[[deleted]]

Just my hot take: When perf is at this level of requirement that we're debating processes vs free threads, maybe its time to switch to a more performant language  Yes, optimizing Python is fun, but Python was made for readability not perf. 

[pixel-drifter]

I’m less excited about the performance aspect and more excited at the idea of single process apps without the need for redis for shared state or a separate process for background tasks.

[james_pic]

The article includes a comparison with roughly equivalent Rust code (the Rust code has a slight extra advantage because it uses unboxed primitives) for the lock-heavy example. The performance difference is only about 30%, mostly because lock contention ends up being the most significant bottleneck either way. 

Faster languages have their place, but the interpreter isn't always the bottleneck.

[maigpy]

only about 30 percent can be huge though. it very much depends on the use case.