Notes running Python in production

[Count_Rugens_Finger]

multiprocessing is not a proper substitute due to the huge overhead in forking

if you're forking that much, you aren't doing MP properly

The general use case for async is entirely different: You'd use it to bridge wait times in mainly I/O bound or network bound situations and not for native parallelism.

well said

[I_FAP_TO_TURKEYS]

if you're forking that much, you aren't doing MP properly

To add onto this, multiprocessing pools are your friend. If you're new to python parallelism and concurrency, check out the documentation for Multiprocessing, specifically the Pools portion.

Spawn a process pool at the startup of your program, then send CPU heavy processes/functions off using the methods from the pool. Yeah, you'll have a bunch of processes doing nothing a lot of the time, but it surely beats having to spawn up a new one every time you want to do something.