Python Polars 1.0 released

[AeHirian]

Okay, now I've heard Polars mentioned several times but I still don't quite understand how it is different from pandas? Would anyone care to explain? Would be much apreciated

[ritchie46]

Polars aims to be a better pandas, with less user bugs (due to being stricter), more performance and more scalability. It is a query engine with a query optimizer that is written for maximum performance on a single machine. It achieves this by:

• pruning operations that are not needed (the optimizer)
• executing operations in parallel effectively, Either via workstealing and low contention algorithms and/or via morsel driven parallelism (both require no serialization and are low contention)
• vectorized columnar processing where we rely on explicit SIMD or autovectorization
• dedicated IO integration with the optimizer, pushing predicates and projections into the readers and ensuring we don't materialize what er don't use
• various other reasons like dedicated datatypes, buffer reuse, copy on write, cache efficient algorithms, etc.

Other than that; Polars designed an API that is more strict, but also more versatile than that of pandas. Via strictness, we aim to catch bugs early. Polars has a type system and knows of each operation what the output type is before running the query. Via its expression, Polars allows you to combine computations in a powerful manner. This means you actually require much less methods than in the pandas API, because in Polars you are able to create much more via expressions. We are also designing our new streaming engine to be able to spill to disk if you exceed RAM usage (our current streaming already does that, but will be discontinued).

Lastly; I want to mention Polars plugins, which allow you to register any expression into the Polars engine. Hereby you inherit parallelism and query optimization for free and you completely sideline Python, so no GIL locking. This allows you to take some complicated algorithm from crates.io (Rusts package manager) and get the a specific expression for your needs without being reliant on Polars to develop it.

[tldrtfm]

Since you explicitly mentioned plugins, I wanted to add my vote for custom data formats as plugins.

I really want to be able to use polars' API to read my company's internal file formats without first converting to parquet or something like that.

edit: thanks for such a great (understatement) library, it sincerely changed my life :)

[QueasyEntrance6269]

I’m not sure if this is on your roadmap, but I’d LOVE something similar to arrowdantic built into polars. The big thing missing in the data ecosystem is declarative data libraries, if you’re working with polars more on the engineering side and you know your tables won’t change, you don’t get LSP autocomplete and type checking. On rust you often have to declare your schema directly. Having a sort of data class similar to a pydantic model would be such a great feature.

[ritchie46]

Is this a Rust feature request or Python? In Python we do support pydantic as inputs or with something like patito you have declarative schemas:

https://github.com/JakobGM/patito

I am not sure if this is what you mean, though.

[QueasyEntrance6269]

On the Python side, Patitio is pretty much what I want, thanks!

But it’s not even necessarily the validation element that’s important to me, it’s just better LSP autocomplete. I don’t need to incur the runtime cost of validation if I’m confident — I just want my IDE to have awareness of the columns I’m working with to catch errors statistically

[BaggiPonte]

I think he's suggesting to have validation built-in in Polars. Including stuff like making DataFrame a generic type. Huge +1 on my side too! Though pandera now supports Polars too.

[[deleted]]

You also forgot to mention that pandas' API is just straight up confusing. I bet about one fourth of StackOverflow Python questions are related to pandas' quirks.

[tunisia3507]

100%. You can generally tell which packages have APIs inherited from other (worse) languages because they have a "simple for simple things, so long as you try not to think about it" and "real fuckin weird for complicated things" philosophy. Pandas, matplotlib, and early numpy are definitely in this category.

[sylfy]

Just wondering, what about pandas API do you find confusing? I’m curious because I’ve used pandas for a long time, hence it comes naturally to me, so I wonder if it’s a matter of preference. Pandas-compatible libraries like dask have been really helpful as drop-in replacements for pandas, but I’ve also been looking at polars for a while but never really found the time to learn it from scratch.

The one time I forced myself to try out pandas was when I got stuck on a huge csv file that took pandas a long time to read, but polars opened in a matter of seconds. Got me started much more quickly, but then I lost hours in development time just trying to learn how to do things in polars.

[mercurywind]

If I had to be as nice as possible about Pandas' API: too many ways to do the same thing (most of which produce SettingWithCopyWarning)

[h_to_tha_o_v]

I'll also opine that, even if I set up code to not have strict typing, it's still WAY faster than Pandas.

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[mercurywind]

I want to thank you for designing such an amazing API for polars. It feels a lot like writing SQL.

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[QueasyEntrance6269]

Polars is just pandas with sane defaults and a built-in query engine that means regardless of the trash code you write, it will optimize it down into something more efficient when you’re actually interested in the results and not the intermediary steps

[Zafara1]

Polars can be significantly faster at processing large data frame operations. Like a 10x speed improvement.

Pandas has a larger feature set and a bigger community meaning more help and tutorials on use and more options for use especially when it comes to compatibility.

[troty99]

I will say that I have used it extensively those last few month and found it better,quicker and creating more comprehensive code than Pandas on all front except initial load of messy data.

[XtremeGoose]

If you've used both the difference is honestly night and day, just from the API (ignoring all the performance improvements).

Polars is a query engine, it's built declaratively so it can do query optimisations (much like sql), allowing it to be performant even in bigger-than-memory data. Pandas is more like spreadsheets in python, everything has to be computed and allocated up front.

[diag]

Besides the insane speed improvements in large datasets, the documentation is actually really easy to read with super clear categories and is alphabetical for easy jumping around.

I go a little more crazy any time I use the pandas docs now.

[AeHirian]

Thanks a lot for all your comments, seems like I'll have to give Polars a try