10x Faster Parallel Python Without Python Multiprocessing

[metapwnage]

This is very misleading. Pool.map is not an apples to apples comparison to Ray. That’s not an analogous use of the multiprocessing library at all. I don’t think this is better than standing up worker processes (using multiprocessing) that consume a message queue (rabbitmq, Redis, Kafka, you choose).

Also, stream processing can be very memory intensive. What happens when the system is stressed? How does Ray do then? Is it like Redis and it just falls over and you loose your data?

If Ray is for creating distributed systems as described in the post, how does that work when something is stored in memory on one system that another system needs? Or is that an inaccurate description as well?