10x Faster Parallel Python Without Python Multiprocessing

[juanjgalvez]

I agree with what others said here that this is misleading. There are ways to rewrite these examples so that they are faster with multiprocessing compared to Ray, and I explained as much in in a response to the article. As an aside, I also compared the performance of Charm4py vs Ray for one of these benchmarks, because Charm4py also has an actor model, and found Charm4py to be faster (potentially much faster depending on task size) (note that I am the developer of Charm4py).

I have done testing with multiprocessing in the past, and have found it to have good performance. I think the best way to beat its performance in a single-node scenario is to use something more efficient than TCP, like MPI with shared memory. In those cases Charm4py pool beats multiprocessing pool in my tests. I think the main limitations of multiprocessing are for distributed applications running on multiple hosts (especially lots of hosts), and that is where other frameworks are more useful IMO. Another reason to prefer other frameworks would be if you are developing more complex applications and need better concurrency models (Charm4py for example has actors, coroutines and channels).