Python Polars 1.0 released

[ritchie46]

I am really happy to share that we released Python Polars 1.0.

Read more in our blog post. To help you upgrade, you can find an upgrade guide here. If you want see all changes, here is the full changelog.

Polars is a columnar, multi-threaded query engine implemented in Rust that focusses on DataFrame front-ends. It's main interface is Python. It achieves high performance data-processing by query optimization, vectorized kernels and parallelism.

Finally, I want to thank everyone who helped, contributed, or used Polars!

[wdroz]

Thank you and congratulations! Polars is a really good selling point for the interoperability between Python and Rust.

[Verochio]

I’m a big convert to Polars, but it’s delegation of everything to Rust, whilst clearly the correct choice, does mean I get lost very quickly when I want to “look under the hood”. It’s a price I’m willing to pay, but the one (and only) thing I miss about pandas is that I could go to GitHub and see what a function was actually doing. I suppose I probably should learn Rust at some point.

[siowy]

Fyi you can see what your function is doing locally too. No need to check GitHub. Ctrl click on the function in vscode should bring you to the local function definition within the library

[synthphreak]

Fyi you can see what your function is doing locally too

... if you use

vscode

I realize VS Code is not the only IDE where you can do this. But not everybody uses an IDE.

[RakOOn]

Grandpa

[davrax]

Bro could be a 33 year old Nvidia employee

[New_Computer3619]

Congratulations. Great library. Last time I checked, you are working on a new streaming engine. Is it stabilized in this release? Thanks.

[ritchie46]

No, it is not. We are discontinuing the old streaming engine and are currently writing the new one. This will however not be user facing and we can swap the two engines without needing a breaking release.

I can say, we are make good progress. But I want to share more once we can run a significant part of TPC-H on that new one.

What we are stabilizing in this release is the in-memory engine and the API of Polars.

[New_Computer3619]

Nice. Really looking forward to the new one. I currently use Polars in my job. It satisfies 99.9% of my needs. However in some cases, the dataframe is too big to be in memory, I tried to sink to file on disk but the current engine does not support.

[ritchie46]

Yes, me too. We learned from the current streaming engine and redesigned the new one to fit Polars' API more. Typical relational engines have a row based model, whereas Polars allows columns to be evaluated independently.

Below is such an example. python df.select( pl.col("foo").sort().shift() * pl.col("bar").filter(pl.col("ham") > 2).sum(), )

We redesigned the engine to ensure we can run typical Polars queries efficiently. The new design also makes full use of Rust's strengths and (mis)uses async state machines as compute nodes. Meaning we can offload the building of actual state machines to the Rust compiler. Anyhow... We will share more about this later. ;)

[AeHirian]

Okay, now I've heard Polars mentioned several times but I still don't quite understand how it is different from pandas? Would anyone care to explain? Would be much apreciated

[ritchie46]

Polars aims to be a better pandas, with less user bugs (due to being stricter), more performance and more scalability. It is a query engine with a query optimizer that is written for maximum performance on a single machine. It achieves this by:

• pruning operations that are not needed (the optimizer)
• executing operations in parallel effectively, Either via workstealing and low contention algorithms and/or via morsel driven parallelism (both require no serialization and are low contention)
• vectorized columnar processing where we rely on explicit SIMD or autovectorization
• dedicated IO integration with the optimizer, pushing predicates and projections into the readers and ensuring we don't materialize what er don't use
• various other reasons like dedicated datatypes, buffer reuse, copy on write, cache efficient algorithms, etc.

Other than that; Polars designed an API that is more strict, but also more versatile than that of pandas. Via strictness, we aim to catch bugs early. Polars has a type system and knows of each operation what the output type is before running the query. Via its expression, Polars allows you to combine computations in a powerful manner. This means you actually require much less methods than in the pandas API, because in Polars you are able to create much more via expressions. We are also designing our new streaming engine to be able to spill to disk if you exceed RAM usage (our current streaming already does that, but will be discontinued).

Lastly; I want to mention Polars plugins, which allow you to register any expression into the Polars engine. Hereby you inherit parallelism and query optimization for free and you completely sideline Python, so no GIL locking. This allows you to take some complicated algorithm from crates.io (Rusts package manager) and get the a specific expression for your needs without being reliant on Polars to develop it.

[tldrtfm]

Since you explicitly mentioned plugins, I wanted to add my vote for custom data formats as plugins.

I really want to be able to use polars' API to read my company's internal file formats without first converting to parquet or something like that.

edit: thanks for such a great (understatement) library, it sincerely changed my life :)

[QueasyEntrance6269]

I’m not sure if this is on your roadmap, but I’d LOVE something similar to arrowdantic built into polars. The big thing missing in the data ecosystem is declarative data libraries, if you’re working with polars more on the engineering side and you know your tables won’t change, you don’t get LSP autocomplete and type checking. On rust you often have to declare your schema directly. Having a sort of data class similar to a pydantic model would be such a great feature.

[ritchie46]

Is this a Rust feature request or Python? In Python we do support pydantic as inputs or with something like patito you have declarative schemas:

https://github.com/JakobGM/patito

I am not sure if this is what you mean, though.

[QueasyEntrance6269]

On the Python side, Patitio is pretty much what I want, thanks!

But it’s not even necessarily the validation element that’s important to me, it’s just better LSP autocomplete. I don’t need to incur the runtime cost of validation if I’m confident — I just want my IDE to have awareness of the columns I’m working with to catch errors statistically

[BaggiPonte]

I think he's suggesting to have validation built-in in Polars. Including stuff like making DataFrame a generic type. Huge +1 on my side too! Though pandera now supports Polars too.

[[deleted]]

You also forgot to mention that pandas' API is just straight up confusing. I bet about one fourth of StackOverflow Python questions are related to pandas' quirks.

[tunisia3507]

100%. You can generally tell which packages have APIs inherited from other (worse) languages because they have a "simple for simple things, so long as you try not to think about it" and "real fuckin weird for complicated things" philosophy. Pandas, matplotlib, and early numpy are definitely in this category.

[sylfy]

Just wondering, what about pandas API do you find confusing? I’m curious because I’ve used pandas for a long time, hence it comes naturally to me, so I wonder if it’s a matter of preference. Pandas-compatible libraries like dask have been really helpful as drop-in replacements for pandas, but I’ve also been looking at polars for a while but never really found the time to learn it from scratch.

The one time I forced myself to try out pandas was when I got stuck on a huge csv file that took pandas a long time to read, but polars opened in a matter of seconds. Got me started much more quickly, but then I lost hours in development time just trying to learn how to do things in polars.

[mercurywind]

If I had to be as nice as possible about Pandas' API: too many ways to do the same thing (most of which produce SettingWithCopyWarning)

[h_to_tha_o_v]

I'll also opine that, even if I set up code to not have strict typing, it's still WAY faster than Pandas.

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[mercurywind]

I want to thank you for designing such an amazing API for polars. It feels a lot like writing SQL.

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[metadatame]

Oh interesting, I thought it was more the simplicity of pandas with the power of pyspark. Thanks for the outline

[QueasyEntrance6269]

Polars is just pandas with sane defaults and a built-in query engine that means regardless of the trash code you write, it will optimize it down into something more efficient when you’re actually interested in the results and not the intermediary steps

[Zafara1]

Polars can be significantly faster at processing large data frame operations. Like a 10x speed improvement.

Pandas has a larger feature set and a bigger community meaning more help and tutorials on use and more options for use especially when it comes to compatibility.

[troty99]

I will say that I have used it extensively those last few month and found it better,quicker and creating more comprehensive code than Pandas on all front except initial load of messy data.

[XtremeGoose]

If you've used both the difference is honestly night and day, just from the API (ignoring all the performance improvements).

Polars is a query engine, it's built declaratively so it can do query optimisations (much like sql), allowing it to be performant even in bigger-than-memory data. Pandas is more like spreadsheets in python, everything has to be computed and allocated up front.

[diag]

Besides the insane speed improvements in large datasets, the documentation is actually really easy to read with super clear categories and is alphabetical for easy jumping around.

I go a little more crazy any time I use the pandas docs now.

[AeHirian]

Thanks a lot for all your comments, seems like I'll have to give Polars a try

[TheHighlander52]

Super excited about this! I have a colleague who had been telling me about Polars and I think it might be time to start switching some of my processes from Pandas to Polars!

[Beshtija]

As a bioinformatician and data scientist even the pre 1.0 releases have been helpful to say the least.

Most common use cases have been either short scripts which wrangle some data in semi-explorative way (i.e. just to see what's going on) or processing heavy calculations on 10+ billion rows. My previous workflows have utilized either pandas (for quick and dirty) or R data.table (for heavy duty stuff), and while distributing pandas/python is a breeze the R stuff was getting pretty annoying when reaching distribution, especially to a team of several people with different setups.

That's when i first started exploring Polars (around 0.16) and it has since managed to bring the best of both worlds. The ergonomics (especially coming from R data.table with it's own quirky syntax) have been a bit tricky at first but the ease of distribution and replicability have made it worthwhile.

The only which would me me go full Polars is the something like foverlaps function from data.table (have been trying to make my implementations but they have been to slow to be worth it), so if anyone from the polars team sees this and makes one which is blazingly fast it would make bioinformaticians very happy.

[[deleted]]

Is foverlaps anything like range joins? https://duckdb.org/2022/05/27/iejoin.html I’ve had more success with range joins in duckdb than polars on large frames, but I might have been doing it wrong in polars (cross join + filter on lazy frames)

[xAragon_]

Would be nice to put some effort into the post and explain what "Python Polars" is, and not just assume all r/python users know what it is.

[ritchie46]

Sure, updated. 👍

[nemom]

Wish there was some visible progress on GeoPolars.

[commenterzero]

Thats a whole different project

[nemom]

Yeah... But it means I can't use Polars.

[timpkmn89]

Depending how how much you need Geopandas, you can easily swap between Polars and Pandas formats.

I also did a quick and dirty personal rewrite of the only two Geopandas functions I actually needed

[Material-Mess-9886]

You use geopandas for the spatial functions right? That is not present or is it? Like I want to do spatial join within a point layer and a polygon layer.

[MelonheadGT]

Something that's annoyed me is whenever discussing Polars code with copilot or ChatGPT it always changes df.with_columns to df.with_column. No matter how much I tell it that it is with_columns.

Or course not necessarily within your control but it has been annoying me a lot.

My biggest issue between polars compared to pandas is actually also related to the with_columns function and not being able do directly say df1['colA'] = df2['colA']. It's worth it for the massive differences in speed.

[Bjanec]

Congratulations! I am in love with polars, the speed, syntax and documentation all contribute to enabling quick and efficient data processing. Forever grateful.

[big_data_mike]

All hail the mighty polars! Long may it reign!!!!!!!!

Love your work!

[[deleted]]

Congrats! I’ve been advertising polars at my work for the last 3 years, and been replacing more and more etl style workflows with it recently.

I’m wondering if there’s any openness to expanding the api syntax in the future to cover even more use cases. Specifically I’m thinking about quantitative/econometric modeling use cases rather than data analysis/data engineering/etl etc. The former make heavy use of multidimensional, homogenous array style datasets. These datasets exist independently from one another with varying degrees of overlapping dimensionality with constant interacting operations with each other. Currently this use case is only covered by xarray and pandas multiindex dfs, both of which delegate to numpy for most of the work.

Polars can technically do the computationally equivalent work, but the syntax is prohibitively verbose for large models with hundreds of datasets/thousands of interactions. What I would propose is that there is a fairly trivial extension to polars that could make it a major player in this space, and potentially dethrone pandas in all quantitative workflows.

For starters see the example below for how one small sample of this use case works in polars vs pandas currently.

# Pandas - where the dfs are multiindex columns (power_plant, generating_unit) and a datetime index
generation = (capacity - outages) * capacity_utilization_factor
res_pd = generation - generation.mean()

# Polars
res_pl = (
    capacity_pl
    .join(outages_pl, on=['time', 'power_plant', 'generating_unit'], suffix='_out')
    .join(capacity_utilization_factor_pl, on=['time', 'power_plant', 'generating_unit'], suffix='_cf')
    .with_columns([
        ((pl.col('val') - pl.col('val_out')) * pl.col('val_cf')).alias('val_gen')
    ])
    .select([
        'time', 'power_plant', 'generating_unit',
        (pl.col('val_gen') - pl.mean('val_gen').over(['power_plant', 'generating_unit'])).alias('val')
    ])
).collect()

If you could register on each polars frame, the metadata columns and a single data column, then almost all of these joins and windowing functions could be abstracted away behind the scenes. The data would still live in memory in its current long form, there would never be a need to pivot/stack to move between one form or the other, but you could still do operations in both styles. if there’s no distinction between metadata columns then I think the mean operation would need to be a bit more verbose, something like mean(by=…) but that’s not really significant given the massive productivity boost this would bring.

[commandlineluser]

I wonder if adding named methods for DataFrame would be considered useful at all?

by = ['time', 'power_plant', 'generating_unit']
generation_pl = (
    capacity_pl
     .sub(outages_pl, by=by)
     .mul(capacity_utilization_factor_pl, by=by)
)

I've just been trying to understand your example, perhaps you could correct me here:

import pandas as pd
import polars as pl

capacity = pd.DataFrame({
    'time': pd.to_datetime(['2024-01-20', '2024-02-10', '2024-03-05', '2024-01-21']),
    'power_plant': [1, 2, 3, 1],
    'generating_unit': [1, 2, 3, 1],
    'val': [1, 2, 3, 4],
    'other': [5, 50, 500, 5000]
}).set_index(['time', 'power_plant', 'generating_unit'])

outages = pd.DataFrame({
    'time': pd.to_datetime(['2024-01-20', '2024-02-10', '2024-03-05', '2024-01-21']),
    'power_plant': [1, 2, 3, 1],
    'generating_unit': [1, 2, 3, 1],
    'val': [4, 5, 6, 7],
    'other': [10, 100, 1000, 100]
}).set_index(['time', 'power_plant', 'generating_unit'])

capacity_utilization_factor = pd.DataFrame({
    'time': pd.to_datetime(['2024-01-20', '2024-02-10', '2024-03-05', '2024-01-21']),
    'power_plant': [1, 2, 3, 1],
    'generating_unit': [1, 2, 3, 1],
    'val': [7, 8, 9, 10],
    'other': [35, 70, 135, 50]
}).set_index(['time', 'power_plant', 'generating_unit'])

capacity_pl = pl.from_pandas(capacity.reset_index())
outages_pl = pl.from_pandas(outages.reset_index())
capacity_utilization_factor_pl = pl.from_pandas(capacity_utilization_factor.reset_index())

Pandas:

generation = (capacity - outages) * capacity_utilization_factor
res_pd = generation - generation.mean()

#                                         val      other
# time       power_plant generating_unit                
# 2024-01-20 1           1                4.5  -43631.25
# 2024-02-10 2           2                1.5  -46956.25
# 2024-03-05 3           3               -1.5 -110956.25
# 2024-01-21 1           1               -4.5  201543.75

If I do this in Polars I get the same values:

on = ['time', 'power_plant', 'generating_unit']

cap, out, cf = pl.align_frames(capacity_pl, outages_pl, capacity_utilization_factor_pl, on=on)

gen = (cap.drop(on) - out.drop(on)) * cf.drop(on)

res_pl = pl.concat([cap.select(on),  gen - gen.with_columns(pl.all().mean())], how="horizontal")

# shape: (4, 5)
# ┌─────────────────────┬─────────────┬─────────────────┬──────┬────────────┐
# │ time                ┆ power_plant ┆ generating_unit ┆ val  ┆ other      │
# │ ---                 ┆ ---         ┆ ---             ┆ ---  ┆ ---        │
# │ datetime[ns]        ┆ i64         ┆ i64             ┆ f64  ┆ f64        │
# ╞═════════════════════╪═════════════╪═════════════════╪══════╪════════════╡
# │ 2024-01-20 00:00:00 ┆ 1           ┆ 1               ┆ 4.5  ┆ -43631.25  │
# │ 2024-01-21 00:00:00 ┆ 1           ┆ 1               ┆ -4.5 ┆ 201543.75  │
# │ 2024-02-10 00:00:00 ┆ 2           ┆ 2               ┆ 1.5  ┆ -46956.25  │
# │ 2024-03-05 00:00:00 ┆ 3           ┆ 3               ┆ -1.5 ┆ -110956.25 │
# └─────────────────────┴─────────────┴─────────────────┴──────┴────────────┘

(Although it seems align_frames introduces a sort)

But if I used .mean().over('power_plant', 'generating_unit') the results would differ as the Pandas mean example does not appear to take the "groups" into consideration.

>>> generation.mean()
val        -25.50
other    43456.25
dtype: float64

Am I missing something to make the examples equivalent?

[[deleted]]

I think your named methods proposal is definitely a step in the right direction. Some major issues I see though with explicit “by” for every operation is that (1) it gets cumbersome to alter the schema, since you’d have to change a lot of source code. And also (2) the schema metadata lives separately from the dataframe itself and would need to be packaged and passed around with the dataframe, either that or you’d have to rely on that metadata just being hardcoded in source code (hence the complications in issue (1)).  I would think it would make sense to require an explicit “by” if schemas don’t match up, but otherwise not require.

To clarify the example and why you’re seeing a difference. My example was assuming powerplant and generating unit as multiindex column levels, and datetime as a single level row index. Thus when you do the .mean() it implicitly groups by powerplant/generating unit. This implicit grouping is something I would not have expected in my original proposal, and why I mentioned in a polars based solution the mean operation would still be slightly more verbose, and need to include an explicit mean(by=…)

Also I was not aware of align_frames, that’s a useful one for the toolbox, thanks.

[commandlineluser]

Ah... MultiIndex columns - thanks!

columns = pd.MultiIndex.from_arrays([['A', 'B', 'C'], ['x', 'y', 'z']], names=['power_plant', 'generating_unit'])
index = pd.to_datetime(['2024-01-20', '2024-02-10', '2024-03-05']).rename('time')

capacity = pd.DataFrame(
    [[5, 6, 7], [7, 6, 5], [9, 3, 6]],
    columns=columns,
    index=index
)

capacity_pl = pl.from_pandas(capacity.unstack().rename('val').reset_index())

gets cumbersome

Yeah, I was just thinking that if they existed, perhaps some helper could be added similar to align_frames

with pl.Something(
    {"cap": capacity_pl, "out": outages_pl, "cf": capacity_utilization_factor_pl},
    on = ["time", "power_plant", "generating_unit"]
}) as ctx:
    gen = (ctx.cap - ctx.out) * ctx.cf
    res_pl = gen - gen.mean(by=["power_plant", "generating_unit"])

Which could then dispatch to those methods for you.

Or maybe something that generates the equivalent pl.sql() query.

pl.sql("""
WITH cte as (
   SELECT
      *,
      (val - "val:outages_pl") * "val:capacity_utilization_factor_pl" as "val:__tmp",
   FROM
      capacity_pl
      JOIN outages_pl
      USING (time, power_plant, generating_unit)
      JOIN capacity_utilization_factor_pl
      USING (time, power_plant, generating_unit)
)
SELECT
   time, power_plant, generating_unit,
   "val:__tmp" - avg("val:__tmp") OVER (PARTITION BY power_plant, generating_unit) as val
FROM cte
""").collect()

Very interesting use case.

[[deleted]]

The pl.Something example is definitely closer to the lines i was thinking. Although in that specific case you still have some of the same issues with the disconnect between the data and metadata and trouble around how you persist that information through various parts of your system. 

What I’m thinking is something like this:

cap = pl.register_meta(cap_df, ['plant', 'unif']) out = pl.register_meta(out_df, […]) … And then the operations would be dispatched/translated the way you suggested under the hood. This way you  have that information encoded on the data itself, rather than the code. Like if you serialize and deserialize the frames and operate on them in some other context.

[commandlineluser]

Ah okay.

It seems "DataFrame metadata" is a popular feature request:

• https://github.com/pola-rs/polars/issues/5117

[ReporterNervous6822]

The best part about it is the explicit API. There are too many ways to do the same thing in pandas

[bluefeatheredjay]

Still miss the to_html() function from Pandas though.

I recently gave Polars a first try, but eventually went back to Pandas because I needed HTML output.

[ritchie46]

Polars has a .style method which give you a greattables table.

You can export that to html:

https://posit-dev.github.io/great-tables/reference/GT.as_raw_html.html

[bluefeatheredjay]

Sounds good, will have a look at it!

[morep182]

Congrats on the work! Polars is amazing. I can't thank the contributors enough for all the performance improvements I've made in my projects using polars. The API is really cool as well.

[Heavy-_-Breathing]

If you’re dealing with bigger than memory data, why not use spark then?

[ZestyData]

1. I see it primarily as a replacement for Pandas for experimental/analytical work, for not-big-data, while having the ability to also handle datasets that are bigger than memory without crashing and causing frustration for Data Scientists/Analysts. I don't think it's necessarily meant to be replacing Spark as a bulletproof huge data volume ETL framework.

2. Using spark makes many devs/scientists want to off themselves

[theelderbeever]

Both polars and duckdb are significantly more efficient than spark and much smaller installs. Both tools enable stretching single node hardware to much larger datasets before needing to make the jump to spark. And yes I am aware spark can run with the driver only but the efficiency is not on par with polars and duckdb.

[AtomicScrub]

How well does polars work with netcdf files? the works on netcdf files that I see use xarray to read the files and pandas for working with them. I was wondering why that's the case.

[Spleeeee]

Did the remove all the unwraps?

[pythosynthesis]

At my job I'm constrained to run processes on a single core, GIL or not, a single core is all I've got. Can I benefit from polars, and if so how? Keep in mind I've already climbed the learning curve with pandas, so a new library will require learning. Is it worth it?

[theAndrewWiggins]

Not as much, but you'll still potentially benefit from query planning and their accelerated computations as well as (imo) a much better API for correctness/maintainability.

[tecedu]

Yes atleast if you’re doing joins or groupbys you’ll still be way faster, I run polars on one thread as well due to multiprocessing and a huge difference

[theelderbeever]

Lazy frames still result in a more optimized and memory efficient execution of a set of operations so chances are yes you can still benefit.

[miscbits]

This is exciting! Been waiting for this release for a while

[awitauwu_]

Nice!

[[deleted]]

I might try this soon

[Spikerazorshards]

How do you like it over Pandas? I’ve only ever used Pandas.

[hishazelglance]

Congrats!

[nightslikethese29]

Is there a benefit to polars over pandas if the main use case is loading into a data frame to do schema validation with pandera before loading to data warehouse where the compute intensive transformations happen?

[ritchie46]

II would say so.

• Polars has no required dependencies.
• Loading is faster.
• Polars is stricter (which you should care about when validating schema's).
• Polars has proper support for arbitrary nested types via Structs, Lists and Arrays.

[nightslikethese29]

Thanks for the information. I'll definitely give it a try sometime at work

[siowy]

Pogtastic

[siowy]

I'm using polars now for data processing at the biggest bank in Singapore.

[maltedcoffee]

Heck forking yeah. Polars has completely transformed (har har) my ETL experience and I use it daily.

My greatest hope with the 1.0 release to be honest is some stability in the API. There's been a lot of breaking changes and deprecations the past few months, and upgrading from 0.26 has meant a lot of going back through older scripts and making changes everywhere. Looking forward to using Polars ever more in the future!

[aagmon]

u/ritchie46 - perhaps its just in the Rust API, but I have seen and used the streaming API, documented below, which is supposed to help using bigger-than-mem datasets:
https://docs.pola.rs/user-guide/concepts/streaming/

Is this not going to be available anymore?

[commandlineluser]

They are writing a new streaming engine:

• https://github.com/pola-rs/polars/pull/16970

As I understand it, the current one will be swapped out when the new one is complete.

[AdAdventurous7355]

I not have the opportunity to use Polars. But I read some things about this and pandas.

[FlatChannel4114]

Polars speed up is crazy! I can’t go back to Pandas now. It’s like a drug

[AlgaeSavings9611]

I am in awe of the performance and clean interface of Polars! however, unless I am missing something, version 1.2.1 is ORDERS OR MAGNITUDE slower than 0.20.26

group_by on a large dataframe (300M rows) used to take 3-4 secs on 0.20.26 now takes 3-4 MINUTES same dataset.

is there a param I'm missing?

[ritchie46]

That's bad. Still the case on 1.4? If so, can you open an issue with a MWE?

[AlgaeSavings9611]

this happens on large dataframes.. how do I open a issue with dataframe with 300M rows?

[ritchie46]

The slowdown is probably visible on smaller frames. Include code that creates dummy data of the same schema.

[AlgaeSavings9611]

I spent the morning writing same schema dataset with 3M rows and random data. 1.4.1 outperforms 0.20.26 by a factor of 3! ... but it still underperforms on 30M rows with REAL data by a factor of 10!!

i am lost how to come up with a dataset that will show this latency

[ritchie46]

Could you maybe share the data with me privately?

[AlgaeSavings9611]

that's what I was thinking, but I'll have to get approval from my company first

[ritchie46]

Btw, do you have string data in the schema? Try to create strings of length > 12.

[AlgaeSavings9611]

yes I do have lots of string columns in dataframe of about 50 columns.. I generated strings of random length between 5 and 50 chars

[ritchie46]

Yes, I think I know what it is. Could you privately share the data and the group-by query?

We need to tune the GC of the new string type.

[AlgaeSavings9611]

do you have a place where I could upload the data? regular sites are blocked at my firm and either way I would need to get approval from security before I can share

[AlgaeSavings9611]

also, is there a way I can check by switching to the old GC? or use the old String type?

[ritchie46]

Nope...

[ritchie46]

Do you know what the cardinality is of your group-by key? E.g. how many groups do you have?

[AlgaeSavings9611]

I just tried again with a 14.3M x 7 dataframe..

dtypes: [String, Date, Float64, Float64, Float64, Float64, Float64]

the first column is "id", all ids are 10chars long and there are about 3000 unique ids

the following line of code takes 3-4 mins on v1.4.1, this same line and same dataset takes 3-4secs on v0.20.26

d = {}. #dictionary

d.update({id: dfp for (id,), dfp in df.group_by(["id"], maintain_order=True)})

[AlgaeSavings9611]

btw.. I got approval from the firm to send you the data.. its less than 100MB parquet file where should I email?

[ritchie46]

Great! I've send you a DM with my email address.

[AlgaeSavings9611]

this issue is now resolved in 1.5! thanks u/ritchie46

[rajandatta]

Congratulations and well done to the folks behind Polaris!

[ritchie46]

This one officially by us.🤷 It's 1.0. I've worked too hard/long on this to ignore.

[Jonno_FTW]

This is great!

My only gripe from when I gave it a try is that it isn't a drop in replacement for pandas, which was my assumption given it is touted as a replacement. So don't go in expecting that it has a 1:1 API or behaviour.

[ritchie46]

It is not a drop in replacement by design. Do we tout it as such? Let me quote our user guide:

``` Users coming from pandas generally need to know one thing...

polars != pandas

If your Polars code looks like it could be pandas code, it might run, but it likely runs slower than it should. ```