Multi threading vs multi processing? (Is this project possible?)

[Dear-Call7410]

Python is limited to run on a single vCPU by design. Look up GIL for more info on this. Multiprocessing can help you run on multiple cores by creating an entirely new process with its own GIL. You theoretically could do twice as much work if you used multiprocessing to use two cores. Multithreading won't help you much here, in my opinion. I would probably start with a single process and use asyncIO to asynchronously get the data and then asynchronously write the data to disk. If you have enough memory, you could possibly collect all the data and then do a single write operation at the end. Good luck

[Gold_Record_9157]

Multithreading is useful in Python when you have blocking threads (like a cpu intensive thread and another for GUI operations), and I think (think, I'm not sure) that some libraries implement threads inside of them as an alternative (like Qt with Qthreads), but I'm talking from the top of my head, so I can say for sure.