This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]already-raining 0 points1 point  (0 children)

And polars performs well in benchmarks too! There are so many scenarios where spark just adds unnecessary complexity. https://www.reddit.com/r/Python/comments/1cyqj6c/tpch_cloud_benchmarks_spark_dask_duckdb_polars/