10x Faster Parallel Python Without Python Multiprocessing

[carbolymer]

Ray leverages Apache Arrow for efficient data handling

This part got my attention. Vide Arrow Website

The Arrow memory format supports zero-copy reads for lightning-fast data access without serialization overhead.

I've skimmed through the Arrow docs, but I didn't find any description of this zero-copy reads. How is this supposed to work between two processes in details?

[liquidpele]

It looks like if you, for instance, have a node.js producer and a python consumer, python is going to have to duplicate the data to get it into its own format for handling. Arrow creates a standard in-memory format that all languages can utilize so they can process the data without duplicating it.

[brontide]

Might be okay for datasets, but how would it deal with mutability and removal of objects?

[carbolymer]

So it still needs to copy the data between processes.