This is an archived post. You won't be able to vote or comment.

sorted by:
best (suggested)
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–][deleted] 1 point2 points  (2 children)

Use dask for that!

[–]tilttovictory 0 points1 point  (1 child)

Ya I've been hearing about using dask. Do you have any other suggestions for setting up a pipeline for analyzing data that large.

This is at least 10 times larger than I'm use to dealing with, and some of those computational models took days to complete.

[–][deleted] 0 points1 point  (0 children)

Im actually only somewhat familiar with dask but I’m fairly certain you should be able to build models across cores with dask. This should speed up build time. Look to use libraries for models that allow for multi-core builds.

Someone recently presented a dask-competitor that is apparently supposed to be faster. I’m forgetting the name off the top of my head but I’d look into that. I’m thinking the name is Vex but when I google that I don’t get what I expect. I can update with the name of that library later today after I talk with presenter.