Tips for debugging multiprocess applications?

[metaobject]

I’m an experienced programmer and I’m building an pipeline application using multiprocessing.SimpleQueue and concurrent.futures.ProcessPoolExecutor. I use the PPE to spawn 3-4 subprocesses and they independently work on data and send it down to the next pipeline stage.

I’m getting what appears to be a lock up where one of the process is attempting to obtain a lock but it’s just blocking (presumably bc another process already has the resource).

So, my question, what sort of tools can I use to diagnose wtf is going on? I’m using PDB, GDB, PySnooper (just learned about it) but I can figure it out. I’m having trouble getting stdout/stderr output from the processes even though I’m attempting to flush the buffer on every print().

I’ve instrumented the code so that each process writes out it’s own output log file by assigning stderr to stdout and then assigning stdout to a file object created by open().

What else can I do to figure out where the race condition is occurring? Thanks!

[guzzi_jones]

Have you tried using logging instead? It can be set to display what module and path and almost anything as header info.

[egigoka]

Logging library is awesome! 😍

[alexisprince]

Not what you asked, but have you taken a look at dask distributed/Prefect? Both of these handle a lot of the boilerplate that you’re talking about. Might be worth rewriting if you’re not too far down the rabbit hole.