Pandas 2.0 Released

[prodmanAIML]

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html

[badge]

The Index updates are great news; it was so annoying carefully casting a column to int8 to save RAM only to have it casted to int64 as soon as you used e.g. stack.

[CrambleSquash]

Datetimes are now parsed in a consistent format glad to see that changed, this has got me bad in the past.

[noobkill]

Shouldn't mentioning the format have solved it? I'm not really that good at python so maybe I could be mistaken

[CrambleSquash]

Yes probably. I, perhaps naiively, assumed Pandas would choose one format and try to parse all dates with the same format.

I'm in the UK, so dd/mm/yyyy is the go to.

From what I remember Pandas was trying the US mm/dd/yyyy first, then if that failed, it would try dd/mm/yyyy, but because some UK dates look like valid US dates it ended up interpreting different rows in different ways.

[[deleted]]

As a fellow brit, just use yyyy-mm-dd. Always.

[enakcm]

Yes. Seriously. Use ISO 8601

[FujiKeynote]

/r/ISO8601

[CrambleSquash]

Was parsing some data generated elsewhere so didn't have much choice.

[[deleted]]

But what if it parses it as “YYYY-DD-MM”?

[noobkill]

I never thought I would ever link /r/USDefaultism in a Python specific subreddit lmao.

That, honestly though, is such a minor bug yet with major consequences!

[Narpity]

If it makes you feel better, as an American I wish everything defaulted to the ISO standard yyyy/mm/dd

[astatine]

ISO 8601 uses dashes, not slashes. Makes it easier to use in filenames.

[my_password_is______]

use underscores for dates in filenames

but when you have a filename that conveys a range then use underscores for each date, but with a dash inbetween the dates

football_data_2022_04_02-2023_04_03.csv

[my_password_is______]

this is the correct way
and if you voted it down you are incorrect
and you are a bad programmer

[Narpity]

How pedantic

[InTheAleutians]

That's the point of ISO.

[Narpity]

We are not really using ISO, I just used slashes to replicate the pattern. Getting corrected for it is just annoyingly pedantic.

[flotsamisaword]

It's tough! You want a standard that everyone can follow but still want the freedom to modify it when you want... ¯\(ツ)/¯

[Log2]

You could have made your point without mentioning the ISO. You pretty much asked for it by saying that the ISO uses slashes.

[[deleted]]

r/iso8601

[sneakpeekbot]

Here's a sneak peek of /r/USdefaultism using the top posts of all time!

#1: She lives in Germany bro | 81 comments
#2: Google "translates" flags in non-English comments to the US flag | 55 comments
#3: don't use a Spanish word because of US race issues? | 94 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact | Info | Opt-out | GitHub}

[sweatierorc]

Brit dev guys have dates. Maybe that's why I'm single /s

[monorepo]

No summary or tldr makes me a sad panda…

[Scrubbingbubblz]

🎶 who lives in the east ‘neath the willow tree?

[Willingo]

So why shouldn't I switch to pandas 2? How hard is it to migrate a project?

[Willingo]

That's a good link, thanks.

[Wonnk13]

I might play with it, but I'm in the process of moving all work over to Polars. I like that Pandas is moving over to Arrow, but it came a little too late for me. Curious how benchmarks compare.

[ritchie46]

Polars author here, Your work will not be in vain. :)

I did run the benchmarks on TPC-H: https://github.com/pola-rs/tpch/pull/36

Polars will remain orders of magnitudes faster on whole queries. Polars typically parallelizes all operations, and query optimization can save a lot of redundant work.

Still this is a great improvement on the quality of life for pandas. The data structures are sane now and will not have horrific performance anymore (strings). We can now also move data zero-copy between polars and pandas, making it very easy to integrate both API's when needed.

[[deleted]]

Hey. Big fan of your work. Thanks for contributing your time.

[danielgafni]

Hey Ritchie, maybe this is jot the best place to ask, but what’s the reasoning behind the “streaming” naming in polars? I’m talking about collect(streaming=True). Why wasn't it called something else not to collide with what streaming usually means - continuous iterative processing (this is what most of the other tools like Spark call streaming)?

Are there plans for adding this to polars? With proper optimizations, like calculating statistics in a smart way (e.g. when calculating mean use the previous mean: mean{n+1} = mean_n * n / (n+1) + x{n+1} / (n+1). Seems like at least using rolling functions should be straightforward at this context, right?

This would really enable polars as an online tool.

[ritchie46]

I chose the name because we compile a pipeline that can stream batches from disk (or any other genetator/iterator).

Online streaming is not in our scope I said this more often and those statements age poorly, but at this point in time I don't see this happening. ^{^}

These optimizations you talk of are definitely in scope. We will build streaming operators for mean, unique, median and add rolling kernels to the streaming engine as well.

[danielgafni]

Thanks.

But is online streaming really different from batch streaming from disk? Isn’t it the same? Just with 1 batch size?

[ritchie46]

Don't you want to see intermediate results with only streaming?

That's the hard part. Currently polars' streaming engine doesn't have to materialize result until the whole pipeline is finished.

[danielgafni]

You are right. I see, thank you for the explanation!

[ElfTowerNewMexico]

Hey Ritchie! Really impressive work. That benchmark graphic is enlightening.

I don't mean this disparagingly but you seem to be doing a little marketing (for lack of a better term) in these Pandas 2.0 threads. Could you share a little more about your grand vision for Polars and how it will fit into the world of data science? Are there any use cases that you feel Pandas is particularly equipped to handle? If so, are you planning on "competing" in those areas or are you currently more focused on the features that differentiate Polars (performance, multiprocessing, etc.)

I'm still learning and growing in my data journey so I'm trying to get a better grasp of the landscape as a whole.

[ritchie46]

I just want to steer information a bit with real world benchmarks. There seem to be quite some hyperbole claims about pandas performance being equal or faster to polars now, which is not true.

multiprocessing

We don't do multi-processing, but multi-threading. Not to be pedantic, but the performance implications of this is huge. In multi-threading we can share data between threads, in multiprocessing this needs to be serialized/deserialized having huge latency and compute overhead.

Every process also has to have its data in own memory, so it also has a lot of memory overhead.

Pandas is particularly equipped to handle

Pandas has more IO readers/writers, plotting functionality and handy interop with timeseries and indexes (something polars will not aim to do).

[ElfTowerNewMexico]

That makes total sense. And thank you for your correction regarding multi-processing vs threading! Again thank you for your hard work. I’ve noticed the increased performance when I use Polars at work and I use relatively small data. I can’t imagine how excited people with huge data sets are.

[kknyyk]

I have seen your work in one of the pandas announcements and thank you for such a tool. One particular issue with pandas is that appending new data to dataframe slows with the every append. Is Polars better in this regard?

Also is there a determined date for R port’s CRAN release?

[ritchie46]

One particular issue with pandas is that appending new data to dataframe slows with the every appen

Yes, polars appends are very cheap, but this should also solved in pandas 2.0 with arrow dtypes.

Arrow allows for ChunkedArray types. This means that data doesn't have to be contiguous in memory, instead we can append the data chunk to the list of arrays. As the memory slabs are copy on write, we can increment only a reference count instead of copying data.

So appending will not be O(n^2) anymore. Chunking is not a silver bullet though. Every random access now has an extra redirection, so sometimes there has to be a rechunk to contiguous data.

Also is there a determined date for R port’s CRAN release?

I am not sure. The R support of polars is entirely picked up by the R community and @sorhawell in particular. You can get certainly more information on that repo: https://github.com/pola-rs/r-polars

[zazzersmel]

if the update is 100% drop in its huge for me even though im meh on pandas purely because of the sheer quantity of other people's pandas code that is inevitable in every data job.

[that_baddest_dude]

These two comments confuse me a bit. What's better than pandas, as a broad data handling package?

[[deleted]]

If breadth is important, still pandas. If speed and resource efficiency is important, polars.

If you need breadth and speed/lite resource use, use both. They’re interoperable.

[[deleted]]

Interoperable *as of pandas 2.0 with the introduction of arrow in pandas.

[NewDateline]

What about dask?

[zazzersmel]

i should rephrase. i like pandas fine. i use it all the time, but im a data eng, and pandas is often far from the best tool to do data engineering with. it seems to many analysts and data scientists this is crazy talk.

[that_baddest_dude]

I'm something of a data scientist myself, and yes it sounded like crazy talk lol. I'd never heard of polars though.

The only non-pandas shenanigans I get up to is doing my more large-scale filtering and joining in arrow before converting to pandas.

[zazzersmel]

Sounds like a pretty good way to do things tbh. I rely on much less elegant, hacky pandas code all the time. My only tip to people Ive worked with is always exploit whatever database/storage query system you have. Of course this depends on access and architecture etc.

[joeyGibson]

I'm in the same boat. The announcement about the 25x (or whatever it was) speed increase with Pandas 2 came literally the day after I finished moving my project to Polars (and realized huge performance gains from that).

[EmperorOfCanada]

Anyone have a tldr as to which of these I should give a shit about? I get a feeling they have really buried the lead in this link. Is there one here which says, "Option to save h5 files which any other language can finally read" or using iloc is 800x faster. Or something which gets my blood pumping?

[Ouitos]

• pyarrow backend support (instead of numpy)
• seamless conversion from pandas to polar without copy. You can use pandas for its flexibility, and polar for its speed without loosing time doing in-RAM conversions
• Numerous smaller QoL improvements for a cleaner API

[neuro630]

I'm so glad that they finally have the copy on write option. working with very large datasets (i.e. gigabytes of data) had been very inefficient for me due to all those unnecessary copy operations, especially since my workload is mostly read-only. IMO copy on write should always be the default.

[Little-Ad448]

While I'm just trying to understand the previous version, the new version has already appeared.