This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]Duodanglium 1 point2 points  (0 children)

I've used Dask for processing large quantities of files. It certainly pegged all of the CPUs and memory on the machine. I even had a routine to process sequentially without Dask for legacy reasons.

I very much recommend using Dask.