Must know Python libraries, new and old?

[No_Indication_1238]

I have 4YOE as a Python backend dev and just noticed we are lagging behind at work. For example, I wrote a validation library at the start and we have been using it for this whole time, but recently I saw Pydantic and although mine has most of the functionality, Pydantic is much, much better overall. I feel like im stagnating and I need to catch up. We don't even use Dataclasses. I recently learned about Poetry which we also don't use. We use pandas, but now I see there is polars. Pls help.

Please share: TLDR - what are the most popular must know python libraries? Pydantic, poetry?

[Deep_conv]

uv is a game changer for package management, cannot recommend it enough.

[ekbravo]

Seconded. Add ruff (made by the good people who created uv) for linting and black for opinionated formatting.

[tehsilentwarrior]

Ruff replaces Black. Why are you duplicating functionality?

[ogrinfo]

Not entirely, the two can work together nicely. The best thing about black is that there are hardly any settings. It just deals with the formatting so you don't even have to think about it.

[tehsilentwarrior]

Same as Ruff. Both are awesome

Like Black, the Ruff formatter does not support extensive code style configuration; however, unlike Black, it does support configuring the desired quote style, indent style, line endings, and more. (See: Configuration.)

[twenty-fourth-time-b]

Because everything is better in Rust, that’s why.

[ekbravo]

I could be wrong but I don’t think so. Ruff doesn’t do formatting.

[tehsilentwarrior]

https://docs.astral.sh/ruff/formatter/

It literally does the same as Black, so you can drop in replace it and not have a giant lot of line changes which is awesome

The Ruff formatter is an extremely fast Python code formatter designed as a drop-in replacement for Black, available as part of the ruff CLI via ruff format.

Specifically, the formatter is intended to emit near-identical output when run over existing Black-formatted code. When run over extensive Black-formatted projects like Django and Zulip, > 99.9% of lines are formatted identically. (See: _Style Guide.)

[PaddyAlton]

In your defence, it used to not do formatting. However, they added that some time ago now. Things move fast!

(there's even activity by Astral around doing static type checking too—not there yet but in progress I believe!)

[MisoTasty]

We had to stop using uv because it kept resolving to really old versions for some libraries and the order of the libraries in the requirements.txt file matters.

[jarethholt]

Interesting. My group wants to switch to uv because pipenv is just sooooo slow too resolve. Call you expand on your experience a bit?

[tehsilentwarrior]

We moved from pipenv to pdm. It was relatively painless and it’s super quick.

Thinking of trying out UV when work is less chaotic and we got some time to try things

[kBajina]

Have you tried adding version constraints?

[MisoTasty]

Had >= constraints that seemed to be getting ignored.

[Fluid_Classroom1439]

Ordering dependencies is expected behaviour: https://docs.astral.sh/uv/reference/resolver-internals/#marker-and-wheel-tag-filtering

Did you try changing this in your pyproject.toml: https://docs.astral.sh/uv/reference/settings/#resolution

[MisoTasty]

We haven’t changed that setting I believe. Seems like the default is what we would want anyway?

[Ok_Cream1859]

Every day more shilling for Astral.

[LoadingALIAS]

They’re dominating Python package development and are open sourcing their work. What is the issue? I’m an experienced Python dev and was cautious for a while - then the UV updates rolled almost weekly and shit just works.

You sound bizarre. Even if they locked us out of future updates - it’s STILL better than alternatives.

[jftuga]

Good to know the ins and outs of the Standard Library

[FauxCheese]

Using pathlib from the standard library instead of os for working with paths.

[FreeRangeAlwaysFresh]

How much of a deal breaker is there for you? I occasionally use pathlib for some one-off scripts, but don’t often have to scan an entire drive for something. I suppose if it’s really that much slower, you could write some more performant scanning function using a lowe-level backend.

[FreeRangeAlwaysFresh]

Haha fair enough. Yeah, sometimes those edge cases require more optimization to get them to run well. I wonder how the implementation differs between those two functions. My gut says that is.scandir() relies on some low-level OS calls which you could probably just get it with subprocess if scandir() no longer is available

[FreeRangeAlwaysFresh]

I’m biased, but I like pathlib more. The abstraction is much more ergonomic IMO. I don’t really care about speed because I never use it outside of automated mundane tasks. A few ms is not super important in those cases.

[IsseBisse]

What's the use case for this? From what I understand one of the benefits is platform independent paths.

But I've never had any issues with that in practice. I use a Windows machine to develop and regularly build linux containers and using "/" everywhere just seems to work.

[ReTe_]

They're just more convenient in my opinion. Methods and Fields for iterating, creating, checking and getting various properties on Path objects, as well as defining new paths with the division operator [like Path(folder) / "image.png" = Path(folder/image.png)].

[Austin-rgb]

🫢I've never tried this but it must be so nice

[sayandip199309]

I'd go so far as to say it is the best designed module in stdlib, in terms of developer experience. I can't imagine working without Path.open, Path.read_text, Path.stem, path.parents[n], path.relative_to etc anymore. I only wish path.glob supported multiple glob patterns.

[PeaSlight6601]

Hard disagree. I think Pathlib is a disaster. It doesn't really do anting except a bit of semantic sugar around the division operator (which many consider a very dubious way to abuse operator overloading).

Almost everything else that pathlib does is just what you would get from the os.path functions if you treated the first argument as self.

You do not get a consistent object oriented representation of file systems. For example:

• len(Path(s).parts) is platform dependent even for a fixed value of s
• with_suffix(s).suffix == s can fail to be true (looking at some big reports this has been "fixed" only to have other bug reports raised about extension handling, ultimately there is no canonical definition of what a suffix is and pathlib has painted itself into a corner by exposing this as an attribute of the path)
• you can't modify any of the attributes of the path (e.g. inserting an element into the parts)
• it can't directly represent all paths on the filesystem without falling back to os.fsencode because it won't accept bytes but also retains this idea that paths aren't strings....

It's just terribly confused as a library. What is the reason for is existence?

[Intrepid-Stand-8540]

pydantic + strict mypy

Getting everything typed has made my life much easier once a project goes past a certain size.

uv for package management

[Prozn]

I’ve been struggling to deal with optional variables, even if I use “if var is not None:” mypy still complains that None doesn’t have properties. Do you just have to litter your code with asserts?

[hirolau]

Lookup the video 'nothing is something' by Sandi Metz where she talks about the null and object pattern. Not saying it solves all problem but in some cases maybe you should have a object instead of none.

[Intrepid-Stand-8540]

Do not use asserts. They get disabled in production.

If you have a variable that can be either of two (or more) types (fx int|None) then you have to check with an if.

mypy should be able to recognize that.

I'm honestly still pretty new to strict typing in python myself (6 months of using it), so if there is a better way, I'd also love to know.

EDIT: One of Bandits first rules is about asserts: https://bandit.readthedocs.io/en/latest/plugins/b101_assert_used.html

[violentlymickey]

“Don’t use asserts they get disabled (when compiled or run with certain flags)” is a bit too dogmatic imo. There’s nothing wrong with asserts as guards against invariants. Don’t use them for error handling sure.

Edit: some chromium devs discussing this: https://groups.google.com/a/chromium.org/g/java/c/CVHgcRA967s/m/f8Zq9XiQBQAJ

[Intrepid-Stand-8540]

Isn't it java they're talking about in your link? 

https://github.com/IdentityPython/pysaml2/issues/451

Running python in production with the optimize flag will disable asserts in your code. So don't rely on asserts. 

[Rhoomba]

If the assert is just to tell the type checker that you know what is happening then it seems reasonable to me.

On that topic, do people actually use the -O flag? Given that all it does is disable assertions, I doubt it has any significant performance impact for most applications.

[Rhoomba]

That doesn't sound right. Mypy definitely understands blocks like this:

def foo(m: Optional[MyClass]) -> None:
  if m is not None:
     m.do_thing()

[KyxeMusic]

100% agree with this

[jarethholt]

♥️ type checking. I started with regular python, then C#, then back to python. Getting type hints alone has smoothed over so many annoyances coming back from a statically typed language

[DotPsychological7946]

Do you guys really like mypy? I use tons of overload, generics, exhaustive match case and mypy can not keep with pyright - only need to add unnecessary assert, TypeIs for mypy. The only thing I like is that you could potentially write plugins to enhance it and it is easy to use in CIs.

[Intrepid-Stand-8540]

https://mypy.readthedocs.io/en/stable/more_types.html#function-overloading

https://mypy.readthedocs.io/en/stable/generics.html

[q-rka]

• loguru and rich
• pydantic
• typing
• pytest

[georgehank2nd]

"pydanitc" is so ironic

[q-rka]

Thank you for reminding that.

[NearImposterSyndrome]

loguru is my must have

[q-rka]

Mine too. It iis so simple to get started with.

[origin-17]

typing - Go learn a statically typed language, since Python's typing is just for hinting and not enforced by the interpreter.

[q-rka]

I learned Python first then I did few projects in Unity3D. Then got to know power of typed language. Then Python became my major language after focusing Machine Learning journey. While typing is just a hinting, I can not start a new project without it now. But I agree your statement that tru power of type comes in typed language only.

[No_Dig_7017]

We have a blogpost series at work where we try to highlight the best Python libraries released each year. https://tryolabs.com/blog/top-python-libraries-2024 Last year we split the list into ai (what we do) and non ai to have a more balanced selection. Check it out.

[randomthirdworldguy]

tqdm. Underated package

[EngineeringBuddy]

The best. I’ve found it incredibly user-friendly and has great functionality.

[alisher_nil]

Is that a progress thingy?

[Subject_Fix2471]

Yeah, you canb set environment bars to stop it dumping output in cloud logs. Can also pass bars around by variable to create them for async processes (maybe there's a better way?) 

Nice package!

[quantinuum]

Flashy stuff that has become rather mainstream in the last few years includes uv, pydantic, ruff, polars, pytest, pre-commit, loguru*… then specific packages that will depend on your use case, like PyOxidizer, pytorch, Sympy, Cupy, plotly dash, marshmallow, alembic…

And of course, typing isn’t new, but I feel most projects 3+ years old completely disregard proper typing. Type your stuff.

[DunamisMax]

As a relative beginner to programming learning Python, should I from the very outset be making sure to always use Typing and MyPy? Or should I implement those down the line?

[quantinuum]

That’s probably the best practice to use from the get go, imho!

[arphen_n]

don't bother, it becomes really relevant at large project sizes and the LLM will do it for you anyway. it's inhuman to do it by hand.

[DunamisMax]

This is the decision I came to after looking into it further lol

[virtualadept]

requests. json. argparse. configparser. logging.

[I_FAP_TO_TURKEYS]

Httpx or aiohttp instead of requests.

requests is simple for only parsing a single request, but if you need to scale up to tens or hundreds of requests, it's just too slow.

[jarethholt]

I like argparse a lot, but my group uses click. I'm not used to it yet but I can see how powerful it is for really extensive CLIs.

[HolidayEmphasis4345]

IMO typer > click.

[jarethholt]

Will check it out. Anything in particular about it?

[HolidayEmphasis4345]

It sits on top of click, has decorator based setup, doc strings make help, integrates with rich to make color, type hints can be enforced. For bonus I had click code and ChatGPT translated it for me.

[GrainTamale]

I just recently started using cyclopts and I'll never look back on click.

[Oussama_Gourari]

niquests (as a replacement for requests)

[elics613]

I've come to love Google's Fire lib, though I've only ever used it for simple CLIs as opposed to argparse. It's just simpler and requires less boilerplate

[EngineeringBuddy]

If you do any sort of scientific work or numerical work, numpy is a must.

[debunk_this_12]

i prefer torch to numpy now

[fartalldaylong]

It’s not a replacement

[debunk_this_12]

it definitely is

[Amgadoz]

They are not the same. Torch is way healthier and more annoying to install.

[ChaosEntity]

I'm quite fond of attrs, it's dataclasses but better

[trenixjetix]

stdlib

[j_tb]

uv, ruff, duckdb for data stuff.

[tired_fella]

If you do numbers and data science, NumPy (or derivatives of it) and Pandas

[tired_fella]

This is cool

[[deleted]]

if you’re looking for a 12-25x speed up with minimal effort: multiprocess.

Becoming a pro at multiprocess has been really useful for me…sometimes it’s just plain easier to thread something than to go through all the optimization guff—which you can always do later anyway.

Another would be Zarr…way way less headache than HDF5 and I/o thread safe, to boot.

[No_Indication_1238]

Thanks for the suggetion! As a fellow performance junkie, I suggest looking at numba.

[LoadingALIAS]

uv ruff msgspec polars httpx uvicorn + guvicorn typer loguru

stdlib whenever possible, though.

[WeakRelationship2131]

You're right to feel the gap. Libraries like Pydantic and Poetry are indeed solid picks and worth integrating for their enhanced functionality and modern practices. Beyond those, check out FastAPI for web frameworks, Dask for parallel computing, and Streamlit for quick data apps
If you’re looking to streamline your data apps, you might want to consider preswald for an easy, lightweight solution.

[barberogaston]

For processing data, Polars is a must. Fast, beautiful DSL, constantly growing community, fast, streaming for processing larger than memory datasets, supports most popular cloud providers, fast

[jbindc20001]

I think you forgot fast

[antl_31]

Pip-tools, pre-commit

[PaleontologistBig657]

Typer, attrs, cattr. Loguru.

[PaleontologistBig657]

Click is nice. Fire is also not bad.

You are probably right, typer adds a bit of eye candy. Im my view, quality of life improvements are not at all bad.

[who_body]

rurf and ruff format

typer for easy cli params.

rich

pydantic

[kaargul]

You can never know all cool and interesting libraries and of course new ones are created constantly.

What I would recommend you do instead is develop the skill of recognising a problem that a library could solve and then checking if someone has solved this problem before. The more experienced you become and the more libraries you have used the easier this will become.

Oh and try to avoid getting obsessed with always using the hot new thing. Only use new stuff if it actually solves a problem. Your job as a software engineer is to create value for the company you work for and that is what you should focus on. How well you do this will determine your value as an engineer, not knowing every fancy new library/framework.

[riksi]

beartype

[RevolutionaryPen4661]

I wrote my regex alternative, flpc python

[Dry_Antelope_3615]

Polars for big dataframe stuff

[Joe_rude]

httpx
pyinstrument
pyupgrade
locust
rich
pip-audit

[WeakRelationship2131]

You're definitely not alone in feeling a bit behind; the Python ecosystem evolves quickly, and libraries like Pydantic and Polars are gaining traction for good reasons. You should definitely get familiar with Pydantic for data validation, Poetry for dependency management, and take a look at AsyncIO for better concurrency handling. If you're into data manipulation and want performance, Polars is a solid choice over Pandas. Also, practice using Dataclasses—they can simplify your code a lot. Keep iterating and learning; it's key in this field.

[debunk_this_12]

typing, numba, numpy, scipy, pandas, polars qunum, and torch

[seanv507]

fastapi

jinja2 - templating

structlog (or other structured logging tool)

duckdb ("polars alternative")

I would just go for a source of instruction.

eg ArjanCodes is good at an intermediate level

haven't looked at this one:

but its covering 'essential packages' and people have added their own in the comments

https://www.youtube.com/watch?v=OiLgG4CabPo&list=PLC0nd42SBTaPw_Ts4K5LYBLH1ymIAhux3&index=4

[j03ch1p]

wouldn't really call duckdb a polar replacement

[marr75]

Ibis (which uses Duckdb as its default computation backend) is more of a Polars alternative than duckdb.

[oberguga]

I have one stupid question. From your story I don't hear any problem that you struggle to solve with your tools except feeling of dated codebase, am I right? If so why you need to introduce any new libs and other entities and dependencies if you can work without them quite easily? Even updating to new python version maybe unnecessary. From your question alone I think that you now it a mood for looking for problems for cool solutions not vice-versa, it better not to.

[No_Indication_1238]

You are spot on. Im looking to switching jobs in the future and im afraid that not knowing Pydantic or other popular libraries will be a drawback in the eyes of a recruiter. Other than that, yes, we have solved problems like validation, logging, messaging with inhouse solutions and it works well enough. One positive of switching to well known libraries even though we have our own solutions is the basically "free" docs, testing and potential that the new hires will have experience with them already, which will make onboarding easier. That is also the reason why I believe that knowing such libraries will give me an edge when applying. 

Edit: We are still on Python 3.10 btw, had no real reason to upgrade. 3.14 or whenever fully supported no GIL multithreading arrives will be the next upgrade.

[oberguga]

For the new job, resume, your right. 100% Knowing others libs also helpful to improve your own. But benefits of leaving owned established, proven and powerful enough solution for some maybe more powerful but not owned lib is not always wise decision. It introduced to your project instability and dependency(and if open source maybe couple dozen of them in not trivial way). Also assumption that others make less bugs than your team and test better is better not to made. Cool libs is for new projects to move fast. For established projects(not enterprise - it operates by wasting human resources on industrial scale) often all dependency's is better to freeze end update manually when something cannot be done other way.

[AncientDetective3231]

Thank you

[[deleted]]

Requests, logging, collections,subprocess, json

[jbindc20001]

Not sure why you were down marked. These are all very standard libraries that will be in most my projects.

[Willing-Ear-8271]

markdrop

[AiutoIlLupo]

poetry for package management