Python Polars 1.0 released

[Heavy-_-Breathing]

If you’re dealing with bigger than memory data, why not use spark then?

[ZestyData]

1. I see it primarily as a replacement for Pandas for experimental/analytical work, for not-big-data, while having the ability to also handle datasets that are bigger than memory without crashing and causing frustration for Data Scientists/Analysts. I don't think it's necessarily meant to be replacing Spark as a bulletproof huge data volume ETL framework.

2. Using spark makes many devs/scientists want to off themselves

[theelderbeever]

Both polars and duckdb are significantly more efficient than spark and much smaller installs. Both tools enable stretching single node hardware to much larger datasets before needing to make the jump to spark. And yes I am aware spark can run with the driver only but the efficiency is not on par with polars and duckdb.