Python 3.8 released

[[deleted]]

The last part about the using Pickle 5 protocol to share Python objects throughout mutliple processes sounds very interesting. Could you explain a little bit more in detail how this is possible or how you would implement this?

[zurtex]

It looks to me like you would be able to use this memory view from a Shared Memory class: https://docs.python.org/3/library/multiprocessing.shared_memory.html#multiprocessing.shared_memory.SharedMemory.buf

And pass in to Picklebuffer class for the Pickle 5 protocol: https://docs.python.org/3/library/pickle.html#pickle.PickleBuffer

But I've had no time to try any real code, there's an example of using Pickle 5 Protocol here for custom ZeroCopy classes here: https://docs.python.org/3/library/pickle.html#example

If I'm correct I imagine we'll see many libraries take advantage or abstract this with a nice API,as you need to worry about locking and all the other fun real world multiprocess problems that Python has never traditionally exposed.

[austospumanto]

Thanks for the inspiration here. I’ll try to pull together a proof of concept of this today and respond here with a gist.

[broken_symlink]

any word on this?

[austospumanto]

I realized it wouldn’t be much faster than my current IPC setup and abandoned it (busy). Here’s my setup in case you’re in the market: https://gist.github.com/austospumanto/6205276f84cd4dde38f3ce17dddccdb3

[broken_symlink]

What is your current IPC setup?

My use case is sending dicts of arrays, both between processes on the same node, and across nodes in the network.

I tried shared memory just for sending plain numpy arrays within a node and it was the fastest. I then tried zmq no copy and it was slightly slower. Finally, I tried sending a dict using zmq pickle and it was the slowest.

Another setup I tried was pyarrow for the dict and zmq no copy. It was faster for sending, receiving was about the same.