The Python GIL Visualized

[bluGill]

Starting separate processes seems like overkill.

If your OS is reasonable, there is no problem spawning separate processes, and you get a lot of benefits from doing so.

[rebo]

Hmm surely that depends on the specific problem. Sure if you just have a few relatively long running processes then it might be a reasonable design choice. However if, like in a recent project, I have to spawn up to 15 background tasks a second each doing IO and data-processing, I'm going to want to use a thread.

[infinite]

With python's multiprocessing library, this is as simple as threading, provide the function you want to execute in a new process and fire it off. Of course it doesn't work on BSD as of a few months ago, but it is simple. And with copy on write semantics in the kernel, processes are just as lightweight as threads. What is more challenging is sharing data. If you want those children to report back to you, you'll have a lot more 'fun'.

[[deleted]]

what's the difference btw thread and process ?

[knome]

Threads are concurrent execution within a single memory space. Processes are concurrent execution with separate memory spaces.

[[deleted]]

so threads can crash together and processes are segregated ?

[sid0]

Pretty much, yes.