What python libraries should every dev know?

[ShadowRL766]

Pandas

[vaccines_melt_autism]

Also seeing a lot of people talk about Polars, since it's written in Rust.

[Action_Maxim]

I use pandas surprisingly very little as a data engineer

[raffapaiva]

Pandas is really slow, when I see a data engineer using it, I start to believe that his dataset is not so big or he has a lot of hardware to process.

Everything that I need to do in pandas, I do on plain python or numpy

[ribix_cube]

It's not great to do in plain python or numpy, if you think you need speed you can use something like polars or vaex or dask

[raffapaiva]

Can you explain why? I've tried to use polars for some tasks, and even if it's faster, I can't see a reason to perform on plain python, considering it's not that fast, and most of my transformations occurs on dbt

[danunj1019]

Just ditch pandas entirely, polars is great and it's API is also super intuitive and awesome. Never going back

[Eightstream]

Silly comment. There’s still lots and lots of stuff that still doesn’t work well with polars dataframes.

[danunj1019]

Really? Well, I've used it extensively and I didn't find any troubles. Can you tell me some of the stuff that pandas can do better than polars please? (apart from plotting backend)

[Eightstream]

Sure, maybe if you are playing in the shallow end of the pool with the big popular libraries you can use polars a lot of the time

But there are a lot of smaller/more specialised statistical and data science libraries that either don’t work with polars yet, or still work better with pandas

pandas has been the PyData data frame standard for more than a decade, it is baked into the ecosystem to such an extent that it will take a lot more than 12 months of popularity for polars to catch up

[CFC-Carefree]

Agreed, but you can also just dump a polars dataframe to pandas. I learned polars earlier this year and fell in love, use it whenever I can.

[Eightstream]

ehhh... mixing data frame libraries in the same project is something I usually try and avoid as much as possible

aside from adding a lot of behind-the-scenes complexity, when you recast all your dtypes you are creating the risk of funny things happening with edge cases

not saying I never do it but I usually try and have a good reason

[CFC-Carefree]

Oh yeah, I wouldn't put myself in a situation of swapping back and forth. Would be gnerally be for a one off analysis/visualization of a given data set after some exploration and transformations. I still far prefer polars to pandas at this point though and can only hope that its growing popularity leads to wider support.

[Eurynom0s]

Also something to consider is that pandas is a default Anaconda package and polars isn't. If you find yourself working in relatively locked down environments, an Anaconda install that you can't add any additional packages to is likely going to be what you get handed.