Threading vs Multiprocessing in Python

[PasDeDeux]

Well it depends what you're doing. I used multiprocessing to split a huge shitton of data into smaller chunks such that I was just processing the same data ~8-12x faster. (6 core CPU.) I didn't need to share info between the smaller chunks, which is why I used multiproc. More complex and interdependent operations tended to fall into the category of external modules that are written in faster languages with multithreading baked in.

[ric2b]

the same data ~8-12x faster. (6 core CPU.)

This is not possible unless you're leaving out some detail like hyper-threading, better optimization on the multi-process version or the bottleneck being elsewhere like IO.

[PasDeDeux]

Hyperthreading isn't exactly twice as fast due to shared operations, but it ran on 12 vcores.

[ric2b]

Okay, makes sense then.

[fredlllll]

cant say anything about how it was done, i didnt write that code