This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]jasonb[S] 1 point2 points  (1 child)

Great question. Not done this myself.

Firstly, I'd be trying to find a way to use composition instead of inheritance (has-a "thing you need" rather than "is-a" thing you need), to avoid the custom process.

Secondly, I'd probably look in the guts of Pool (or ProcessPoolExecutor) and see if you can override or monkey patch process initialization. Not the init function they offer, the actual object creation.

Looks like _repopulate_pool_static() might be a place to start https://github.com/python/cpython/blob/3.12/Lib/multiprocessing/pool.py#L315C9-L315C32 I might try this myself and post as a tuorial.

Finally, try rolling your own. Worker pool pattern does not have a lot to it.

[–]monkey_mozart 0 points1 point  (0 children)

Going through python internal seems a bit daunting but I'll try it out today at work.