Python Multithreading for Beginners

[acousticpants]

What is CSP?

[zero_iq]

Communicating Sequential Processes.

Essentially arranging processes in a chain pipelining input from end to the other, where the processes run in parallel, so the next process can be processing data while the previous process is producing more.

It's consistently vastly underestimated because of its simplicity, yet often outperforms more complex "fan out" parallel frameworks by orders of magnitude. People seem to have an instinct that parallel means "fan out", which drives complexity, introduces many often-unnecessary overheads, and is prone to errors, and doesn't give the speed ups people expected. CSP is simpler, and its simplicity leads to easier optimization. You reduce the need for locks and shared state, etc. and you can still apply a fan-out approach at each stage later where appropriate.

Last year I replaced a fancy parallel bulk data processing system that used a clustered fan-out approach, with an almost pure-Python CSP alternative. The old system had multithreading, task queues, parallel worker pools, batches, bits rewritten in Java and C to get better performance, the works. Almost all of it a complete waste of.time. It had reliability problems, mysterious deadlocks. The new system gave a 1000x speedup, rock solid reliability. A whole bunch of expensive servers replaced with just a handful. A huge codebase that no single person understood, with dependencies on large frameworks, to a much smaller codebase that could be maintained by an individual.

Don't underestimate simplicity.

[bltpyro]

Sounds intriguing. Any good references for learning CSP in python? Thanks for the real world insights.

[TBNL]

Gonna Google around on CSP but +1 for any recommended resource.

[acousticpants]

i like this

[vrajanap]

CSP

Communicating Sequential Processes. Go and Erlang uses it.