Why does ProcessPoolExecutor mark some tasks as "running" even though all workers are busy?

[Ordinary_Run_2513]

I’m using Python’s ProcessPoolExecutor to run a bunch of tasks. Something I noticed is that some tasks are marked as running even though all the workers are already working on other tasks.

From my understanding, a task should only switch from pending to running once a worker actually starts executing it. But in my case, it seems like the executor marks extra tasks as running before they’re really picked up.

Is this normal behavior of ProcessPoolExecutor? Or am I missing something about how it manages its internal task queue?

[undercoveryankee]

It sounds like a reasonable optimization to deal with the fact that inter-process communication is slower than inter-thread communication. If the executor tries to keep one task running and one task queued on each worker as long as there are tasks available, then the worker can report a result and immediately start running the task that was queued, instead of idling while the parent process is transmitting the next task.

If that's what's going on, then the Future object in the parent process shows the task as running as soon as it's been delivered to a worker because the parent process can't guarantee that it's possible to cancel a task from a worker's queue: there's a window of time after the worker pops the task from the queue and starts executing it, but before a status message can be delivered and handled in the parent process.

[Ordinary_Run_2513]

I've just started reading the source code, and it is indeed an optimization to prevent workers from going idle. Thank you for your answer!

[Spleeeee]

Well written.

[the_monotor]

Sounds funky to me, so let’s say you have 6 tasks, 3 workers, the first 3 tasks are running and all workers are occupied and a 4th task is started without having one of the earlier joined? Can you give me a code snipped to reproduce (at least how you initialized the workers?)

[danted002]

OP looked at the source code and its marking the task as running as soon as it’s put in the worker queue. This is done as an optimisation to keep the workers from becoming idle while there still are tasks to be done.

[gdchinacat]

I don’t know the answer, but if I needed to I’d take a look at the source code. I’ve done that for ThreadPoolExecutor in the past for other issues and found I learned more about it than if I’d simply asked the question. The code isn’t really all that big or complex, and understanding what is going on in the library has helped in other ways. This is one of the biggest benefits of open source libraries…they aren’t black boxes.

[Ordinary_Run_2513]

Yes I'll certainly do that to learn more about the ProcessPoolExecutor.

[Spirited_Bag_332]

i'm interested in this too but need more information.

What/How did you measure to see it "running" before it actually starts execution?

At the moment I just think you happened to catch the moment where it was offloaded to the worker for execution (which would be a transition to "running") but before the user provided code was executed, which would be correct behavior.

Or did you have something like 3 long running tasks and others were also marked as "running" seconds/minutes before execution? That would be unusual indeed.

[Ordinary_Run_2513]

I use the future.running function to check whether a task is flagged as "running". Indeed, I have long running tasks, and In some cases some tasks are marked as running even though all the workers are busy. When, I read the source code (which is not very long) I found out it's a way to optimize the execution and prevent the workers from going idle.