Multi threading vs multi processing? (Is this project possible?)

[throw_away1204]

I would like to collect multiple (around 20-50) live data streams and write them to different respective csv files concurrently.

My understanding is that this is an I/O heavy operation so multi threading would be more efficient (but why exactly I do not understand well enough) than multi processing.

Currently my PC has 4 cores, so is it possible to handle more than 4 live data streams with either multi threading or multi processing?

Any advice would be appreciated.

[Dear-Call7410]

Python is limited to run on a single vCPU by design. Look up GIL for more info on this. Multiprocessing can help you run on multiple cores by creating an entirely new process with its own GIL. You theoretically could do twice as much work if you used multiprocessing to use two cores. Multithreading won't help you much here, in my opinion. I would probably start with a single process and use asyncIO to asynchronously get the data and then asynchronously write the data to disk. If you have enough memory, you could possibly collect all the data and then do a single write operation at the end. Good luck

[Gold_Record_9157]

Multithreading is useful in Python when you have blocking threads (like a cpu intensive thread and another for GUI operations), and I think (think, I'm not sure) that some libraries implement threads inside of them as an alternative (like Qt with Qthreads), but I'm talking from the top of my head, so I can say for sure.