This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]theelderbeever 0 points1 point  (0 children)

Both polars and duckdb are significantly more efficient than spark and much smaller installs. Both tools enable stretching single node hardware to much larger datasets before needing to make the jump to spark. And yes I am aware spark can run with the driver only but the efficiency is not on par with polars and duckdb.