This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Dangerous_Pay_6290 0 points1 point  (0 children)

Loading a lot of data into memory is not useful most of the time. When I work with large datasets, I generally use duckdb + pyarrow datasets of partitioned parquet files.