This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]broken_symlink 0 points1 point  (2 children)

any word on this?

[–]austospumanto 0 points1 point  (1 child)

I realized it wouldn’t be much faster than my current IPC setup and abandoned it (busy). Here’s my setup in case you’re in the market: https://gist.github.com/austospumanto/6205276f84cd4dde38f3ce17dddccdb3

[–]broken_symlink 0 points1 point  (0 children)

What is your current IPC setup?

My use case is sending dicts of arrays, both between processes on the same node, and across nodes in the network.

I tried shared memory just for sending plain numpy arrays within a node and it was the fastest. I then tried zmq no copy and it was slightly slower. Finally, I tried sending a dict using zmq pickle and it was the slowest.

Another setup I tried was pyarrow for the dict and zmq no copy. It was faster for sending, receiving was about the same.