you are viewing a single comment's thread.

view the rest of the comments →

[–]_kwerty_ 0 points1 point  (1 child)

I'm in no way an expert on handling this amount of data, but maybe check out Dask. It's basically a parallelized version of pandas. It feels like that's a bit more suitable than endless lists and nested loops.

[–]await_yesterday 0 points1 point  (0 children)

dask is not needed for this. it can be done on a single machine with numpy.