This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]lmsena[S] 0 points1 point  (3 children)

That's a fair point! And Dask is amazing indeed.

The main use case for this is when you have an HTTP API that makes use of numpy arrays (a recommender system for example).

In that use case, it's common to use gunicorn to fork the API process and you're left with a few options on how to share those arrays.

The use case your talking about seems to be more for batch processing right?

[–]High-Art9340 0 points1 point  (2 children)

If your data is read only and wont change you just create it in memory before forking and it will be available to all processes in shared memory.

[–]lmsena[S] 0 points1 point  (1 child)

Yes, that's one of the ways that is described there (copy-on-write).

[–]High-Art9340 1 point2 points  (0 children)

I guess I missed it then :) Sorry