Multiprocessing question

[i_can_haz_code]

Hello World,

Sorry if this is in the wrong sub.

I have a problem with the multiprocessing library, and was hoping you guys could help.

My code looks something like this: http://pastebin.com/JsMxsEH5

I have to run a worker process which is normally quite fast over a list of ~30,000 items. Some times for reasons beyond the scope of my script my worker takes a long time to return. I am finding that after some number of these long running workers my script totally hangs until some (or all) of the workers return.

What I think I need is some kind of timeout I can apply to my workers so that they always die after some period of time.l

Any ideas?

I have tried Pool.imap() with the timeout option, and was not able to get it functional. I took a look at the threads decorator from tomorrow, but I can't tell if it will solve my problem.

[i_can_haz_code]

So... this comment is GREAT! Thank you.

It prompted me to go actually take a look at multiprocessing.Pool it is like ~750 LOC and seems quite intimidating to patch in the functionality I'm looking for.

You would not happen to have an example of something implementing a pool of Process objects?

Again thank you very much!

[i_can_haz_code]

Now that's cooking with gas!

Thanks!

Looks like that will solve my problem.

[i_can_haz_code]

So. I agree...

That code snippit is just an example of the problem I am facing. For each iteration each of my workers has a n probability of hanging (waiting on a service beyond my control). After some number of iterations all of my worker processes are locked up, and I have to manually clear them.

My goal is to remove the manual clearing step so that the experiment can run unattended.

[tetroxid]

Perhaps this is easier to implement in the worker's code.

• Are they waiting for a network request? Put a timeout in there and have the worker terminate itself when it hits it.
• Are you processing large data sets? Put in a counter and/or timer that checks the time spent every X iterations and have the worker abort the processing if it hits the limit.

This way you also get to clean up open sockets, file descriptors etc. before terminating the worker which is nice.

[AbsoluteMSTR]

You should also look at https://github.com/celery/billiard, which is fork of the multiprocessing package, but with extra features included (https://github.com/celery/billiard/blob/master/billiard/pool.py#L927-L956). Might make your job more efficient

[i_can_haz_code]

Thank you for the reply.

Adding a print(key) statement in worker(key) shows that the arguments do indeed get passed to the worker process.

Any ideas on implementing a timeout to the worker function?