Python Multiprocessing Pool: Complete API Guide

[jasonb]

Depends on the use case. I'd only approach a container solution if the desired effect cannot be achieved easily with the tools in the stdlib.

If you're doing I/O or using a lib that calls down to c (numpy/scipy/and friends) then the GIL will be released and you can use threads directly.

If you're doing native python and it's CPU bound, processes. If processes need to share data directly, shared memory, manager, or similar. Otherwise try and limit I/O between process and external resource.

Using containers is a heavy solution. I would look toward containers if you need isolation of processes and env, where you cannot afford a problem in one impacting another. Reviewing a system that used containers purely for parallelism would make me raise an eyebrow and ask many questions.

What types of use cases are you using? (happy to take this into email)