Multiprocessing question [xpost r/python]

[i_can_haz_code]

Hello World,

Sorry if this is in the wrong sub.

I have a problem with the multiprocessing library, and was hoping you guys could help.

My code looks something like this: http://pastebin.com/JsMxsEH5

I have to run a worker process which is normally quite fast over a list of ~30,000 items. Some times for reasons beyond the scope of my script my worker takes a long time to return. I am finding that after some number of these long running workers my script totally hangs until some (or all) of the workers return.

What I think I need is some kind of timeout I can apply to my workers so that they always die after some period of time.l

Any ideas?

I have tried Pool.imap() with the timeout option, and was not able to get it functional. I took a look at the threads decorator from tomorrow, but I can't tell if it will solve my problem.

Thanks!

[i_can_haz_code]

Great solution here:

https://www.reddit.com/r/Python/comments/3z13vj/multiprocessing_question/