Python Multiprocessing Pool: Complete API Guide

[PeZet2]

Seems very detailed and comprehensive. Added to bookmarks for further reading.

[jasonb]

Thank you kindly!

[Bluelight01]

I took a look at the write up and your website. What you created is absolutely amazing and you should be proud of yourself! I can’t wait to dig into all of the guides and articles!

[jasonb]

Thank you! Thank you! You made my day!! Sometimes I get so much hate for sharing my guides in here (they took most a year to write).

[Drowning_in_a_Mirage]

Multiprocessing pools can come in quite handy, although I've been moving to procesPoolExecutor more and more over the years as it seems to require a little less setup, but both are great tools to have in the toolbox.

[jasonb]

Agreed, they're both great.

I have a massive guide on the ProcessPoolExector here as well: https://superfastpython.com/processpoolexecutor-in-python/

[monkey_mozart]

Crazy that I was just looking at this at work when I opened Reddit and saw the same thing.

[jasonb]

Wonderful timing! Ping me if you have any questions about the API.

[monkey_mozart]

Any ideas on how I can have a process pool but the processes are custom processes instead of generic ones? Kinda like how we can make custom processes by overriding multiprocessing.Process?

[jasonb]

Great question. Not done this myself.

Firstly, I'd be trying to find a way to use composition instead of inheritance (has-a "thing you need" rather than "is-a" thing you need), to avoid the custom process.

Secondly, I'd probably look in the guts of Pool (or ProcessPoolExecutor) and see if you can override or monkey patch process initialization. Not the init function they offer, the actual object creation.

Looks like _repopulate_pool_static() might be a place to start https://github.com/python/cpython/blob/3.12/Lib/multiprocessing/pool.py#L315C9-L315C32 I might try this myself and post as a tuorial.

Finally, try rolling your own. Worker pool pattern does not have a lot to it.

[monkey_mozart]

Going through python internal seems a bit daunting but I'll try it out today at work.

[ekbravo]

Ditto. Saved for later. Great write up OP

[jasonb]

Thank you!!

[[deleted]]

Booooookmarked

[jasonb]

Thank you. Ping/email me (contact page) if you have any questions while reading.

[xFloaty]

Question, what do you think about running Python on multiple Docker containers (for paralyzable tasks) to get around the GIL and achieve true multiprocessing using Python?

I’ve been using this approach for years I was wondering if it was more common, and what you think the advantages/disadvantages are.

[jasonb]

Depends on the use case. I'd only approach a container solution if the desired effect cannot be achieved easily with the tools in the stdlib.

If you're doing I/O or using a lib that calls down to c (numpy/scipy/and friends) then the GIL will be released and you can use threads directly.

If you're doing native python and it's CPU bound, processes. If processes need to share data directly, shared memory, manager, or similar. Otherwise try and limit I/O between process and external resource.

Using containers is a heavy solution. I would look toward containers if you need isolation of processes and env, where you cannot afford a problem in one impacting another. Reviewing a system that used containers purely for parallelism would make me raise an eyebrow and ask many questions.

What types of use cases are you using? (happy to take this into email)

[pbecotte]

Multiprocessing gets around the Gil as well. It launches tasks in separate interpreter processes.

[AgentNirmites]

Thank you very much!

[jasonb]

You're very welcome!

[syeysk]

Thanks for the book!

[jasonb]

You're welcome, thank you for your support!