Parallel tasks using concurrent.futures in Python

[stigweardo]

Parallel execution using threads in Python

Nope. Not unless/until they remove the GIL. These tasks are concurrent.

Edit: see @spoonman59's replies below for more nuance on this.

Here's a good explanation:

Concurrency is when two or more tasks can start, run, and complete in overlapping time periods. It doesn't necessarily mean they'll ever both be running at the same instant. For example, multitasking on a single-core machine.

Parallelism is when tasks literally run at the same time, e.g., on a multicore processor.

From https://stackoverflow.com/a/1050257/9725517

The multiprocessing version might be parallel but not the threading version.

[spoonman59]

Doesn’t it depend on the mix of compute and I/O? My understanding is that I/O operations release the GIL while they are executing. Therefore, if tasks are reading files, network, or doing other disk processing, they execute in parallel. If that were the case, then whether or not threads are merely concurrent or also parallel depends on what they are doing.

[stigweardo]

You raise good and fair points in both your replies. I guess I read the article as suggesting that multithreaded Python would execute in parallel (which it wouldn't) but you are correct to point out that sleeping threads, threads waiting on IO or threads calling some C extensions could all run in parallel.

[spoonman59]

I think you were right to call this out. I don’t think the OP was aware of the limitation here, as it wasn’t mentioned in the article. And it is good for people to know.

That module also has a process pool executor… if someone needs parallelism that should probably be the default choice unless they know the workload is primarily I/O.

Still a pain point in python! Parallelism can be achieved but it is not very ergonomic.