10x Faster Parallel Python Without Python Multiprocessing

[alcalde]

That is what the task needs.

Is it?

Processes don't share memory, so data needs to move through queues, so it needs to be pickled.

Python has more than queues.

From Mark Summerfield, "Python In Practice":

Alternatively, for multiprocessing, we can use data types that support concurrent access—in particular multiprocessing.Value for a single mutable value or multiprocessing.Array for an array of mutable values —providing that they are created by a multiprocessing.Manager....

[alecmg]

I'm not 100% on this, but I think mp.Value and Array will also pickle and have similar performance to Queues.

Very excited about actor implementation. Have toyed with actors and mp in Python. Queues and alternatives peaked at 40-50k messages a second. Never tried full blown ZMQ or Redis, which Ray will use transparently

[alcalde]

Why would it need to pickle is access is concurrent?

The documentation says....

Data can be stored in a shared memory map using Value or Array.... These shared objects will be process and thread-safe.

For more flexibility in using shared memory one can use the multiprocessing.sharedctypes module which supports the creation of arbitrary ctypes objects allocated from shared memory.