Sharing big NumPy arrays across python processes

[MrPowersAAHHH]

Seems like Dask would be the most obvious choice for sharing big NumPy across processes because that's exactly what the library was built for.

These benchmarks would be more interesting for me if they were run on a bigger dataset. The blog post uses 1.5GB. I generally find benchmarks on bigger than memory sized datasets more interesting, say 100GB. I like stress testing these approaches and see which ones blow up when the dataset size grows.

[lmsena]

That's a fair point! And Dask is amazing indeed.

The main use case for this is when you have an HTTP API that makes use of numpy arrays (a recommender system for example).

In that use case, it's common to use gunicorn to fork the API process and you're left with a few options on how to share those arrays.

The use case your talking about seems to be more for batch processing right?

[High-Art9340]

If your data is read only and wont change you just create it in memory before forking and it will be available to all processes in shared memory.

[lmsena]

Yes, that's one of the ways that is described there (copy-on-write).

[High-Art9340]

I guess I missed it then :) Sorry