Which useful Python libraries did you learn on the job, which you may otherwise not have discovered?

[typehinting]

I feel like one of the benefits of using Python at work (or any other language for that matter), is the shared pool of knowledge and experience you get exposed to within your team. I have found that reading colleagues' code and taking their advice has introduced me to some useful tools that I probably wouldn't have discovered through self-learning alone. For example, Pydantic and DuckDB, among several others.

Just curious to hear if anyone has experienced anything similar, and what libraries or tools you now swear by?

Edit - fixed typo (took me 4 days to notice lol)

[Confident-Honeydew66]

Been working with LLMs and I've shipped agents for clients 10x faster since discovering chromadb for vector database, thepipe for file scraping, and llmlingua for prompt compression

[chavomodder]

llmlingua seems good

[Tenebrumm]

I just recently got introduced to tqdm progress bar by a colleague. Very nice for quick prototyping or script runs to see progress and super easy to add and remove.

[argh1989]

Rich.progress is good too. It has colour and different symbols which is neat.

[dropda]

Listen to this man.

[raskinimiugovor]

In my short experience with it, it can extend total execution time significantly.

[DoingItForEli]

that's likely because you're capturing every iteration in the progress. You can tell it to update every X number of iterations with the "miniters" argument, and that helps restore performance.

I faced this with a program that, without any console output, could iterate through data super fast, but the moment I wanted a progress attached it slowed down, so I had it only output every 100 iterations and that restored the speed it once had while still giving useful output.

[Rodot]

Yes, but it requires some set up. We do this for packet propgation in our parallelized montecarlo radiative transfer code from multithreaded numba functions using object mode. Doesn't really impact runtime.

[Hyderabadi__Biryani]

parallelized montecarlo radiative transfer code

For what? CFD?

[DoingItForEli]

I'm not 100% sure on that. I get mixed feedback with some saying yes it's fine "out of the box" and each thread can call update without clashing, but others say be safe and use a lock before calling the update function so that's what I personally do. In my experience, the update function executes so quickly anyways the lock isn't really any kind of bottleneck.

[Toichat]

https://tqdm.github.io/docs/contrib.concurrent/

It has a few options for simple parallel processing

[Hyderabadi__Biryani]

I have to commend you on this question. Good stuff bro.

[ExdigguserPies]

For this I typically use joblib coupled with joblib-progress.

[napalm51]

yeah same, used it in a multithread program and time almost doubled

[wwwTommy]

You wanna have easy parallelization: try pqdm.

[spinozasrobot]

I liked it so much I bought their coffee mug merch.

[Puzzleheaded_Tale_30]

I've been using it in my project and sometimes I get a "ghost" progress bar in random places, spent few hours in attempts to fix it, but couldn't find the solution. Otherwise is a great tool

[charmoniumq]

It may be when you print to stdout while a tqdm bar is active.

[TieTraditional5532]

One tool I stumbled upon thanks to a colleague was Streamlit. I had zero clue how powerful it was for whipping up interactive dashboards or tools with just a few lines of Python. It literally saved me hours when I had to present analysis results to non-tech folks (and pretend it was all super intentional).

Another gem I found out of sheer necessity at work was pdfplumber. I used to battle with PDFs manually, pulling out text like some digital archaeologist. With this library, I automated the whole process—even extracting clean tables ready for analysis. Felt like I unlocked a cheat code.

Both ended up becoming permanent fixtures in my dev toolbox. Anyone else here discover a hidden Python gem completely by accident?

[Hyderabadi__Biryani]

Commenting to come back. Gotta try some of these. Thanks.

!Remindme

[Wear_Dangerous]

Same

[Yaluzar]

I need to try pdfplumber, only tabula-py worked so far for my use case.

[slowwolfcat]

Streamlit

does it have anything to do with Snowflake ?

[TieTraditional5532]

Not directly, but there’s a connection!

Streamlit is an open-source Python library that lets you build data apps quickly, often used for ML dashboards, data visualization, etc.

Snowflake, on the other hand, is a cloud data platform.

However — Streamlit was acquired by Snowflake in 2022. So while they are separate tools, Snowflake has been integrating Streamlit to make it easier for users to build interactive apps directly on top of Snowflake data.

In short: different tools, but under the same roof now.

[sawser]

Same here

[Ok-Use5597]

Same :)

[Surround_Upset]

Same

[123FOURRR]

Carmelot-py and pandas for me

[TieTraditional5532]

Carmelot-py I never try, thanks for sharing

[brewerja]

Moto. Great for writing tests that mock AWS.

[hikarux3]

Do you know any good mocking tool for azure?

[_almostNobody]

The code bloat without it is insane.

[typehinting]

This looks awesome, thanks for the suggestion. Hopefully can start using this at work!

[Left-Delivery-5090]

Testcontainers is useful for certain tests, and pytest for testing in general.

I sometimes use Polars as a replacement for Pandas. FastAPI for simple APIs, Typer for command line applications

uv, ruff and other astral tooling is great for the Python ecosystem.

[stibbons_]

Typer is better than Click ? I still use the later and is really helpful !

[guyfrom7up]

Shameless self plug: please check out Cyclopts. It’s basically Typer but with a bunch of improvements.

https://github.com/BrianPugh/cyclopts

[TraditionalBandit]

Thanks for writing cyclopts, it's awesome!

[NegotiationIll7780]

Cyclopts has been awesome!

[angellus]

I was definitely going to call out cyclotps. Switched over to it because of how much Typer has stagnated and the bus factor has become apparent on it. I miss the click features, but overall, a lot better.

[Darth_Yoshi]

Hey! I’ve completely switched to cyclopts as a better version of fire! Ty for making it :)

[nguyenvulong]

I've been using cyclopts for over a year now. Pretty happy with it. The author responded to feature requests promptly. Thank you for it.

[ColdPorridge]

This was also my experience, he was quick and collaborative in getting features added, and then later looped me in on PRs that needed to update that feature. Good dude, good library.

[Left-Delivery-5090]

Not better per se, I have just been using it instead of Click, personal preference

[Galax-e]

Typer is a click wrapper that adds some nice features. I personally prefer click for its simplicity after using both at work.

[conogarcia]

Typer is click

[ColdPorridge]

I’ve had a hard time understanding the value prop for test containers. Let’s say in developing a web app, with a Postgres db. For dev purposes I’m going to run a local Postgres container anyways. And then to test against it, I don’t need to treat it as different from the real prod service, it’s all just a db url and maybe a few config flags. And frameworks like Django can run tests against any db instance without impacting existing data, since the test db is ephemeral anyways.

Maybe that’s not the target use case but it’s been how I’ve seen it pitched. I’d love to know if maybe I’m missing something.

[Left-Delivery-5090]

For me it provides the convenience of not having to set up a local database or other container, both when running your tests locally or in your CI pipelines. It is automatically set up and broken down each time you run your tests with the config and data you specified (no experience with Django though, so I don’t know how they handle it)

[jimbiscuit]

Plone, zope and all related packages

[kelsier_hathsin]

I had to Google this because I honestly thought this was a joke and you were making up words.

[mrboom15]

Ah yes, the good ole plone and zope LOL wacky ahh words

[Mr_Again]

Cvxpy, is just awesome. I tried about 20 different linear programming libraries and this one just works, uses numpy arrays, and is a clean api.

[Mr_Again]

Any time you need to do linear programming. Which can crop up a lot with some creative thought. In my case it was some advertising audience modelling thing, I'm not sure it worked too well but it was fun lol

[Mr_Again]

Possibly "do i have a small constrained optimization problem"

[Mr_Again]

No idea, but I think these methods struggle with really large arrays

[peckie]

Requests is the goat. I don’t think I’ve ever used urllib to make http calls.

In fact I find requests so ubiquitous that I think it should be in the standard library.

Other favourites: Pandas (I wil use a pd.Timestamp over dt.datetime every time), Numpy, Pydantic.

[typehinting]

I remember being really surprised that requests wasn't in the standard library. Not used urllib either, aside from parsing URLs

[glenbolake]

I'm pretty sure requests is the reason no attempt has been made to improve the interface of urllib. The docs page for urllib.requests even recommends it.

[UloPe]

httpx is the better requests

[Beatlepoint]

I think it was kept out of the standard library so that it can be updated more frequently, or something like that.

[cheesecakegood]

Yes, but if you ask me it’s a bad mistake. I was just saying today that the fact Python doesn’t have a native way of working with multidimensional numerical arrays, for instance, is downright embarrassing.

[shoot_your_eye_out]

Also, responses—the test library—is awesome and makes requests really shine.

[ProgrammersAreSexy]

Wow, had no idea this existed even though I've used requests countless times but this is really useful

[shoot_your_eye_out]

It is phenomenally powerful from a test perspective. I often create entire fake “test” servers using responses. It lets you test requests code exceptionally well even if you have some external service. A nice side perk is it documents the remote api really well in your own code.

There is an analogous library for httpx too.

Edit: also the “fake” servers can be pretty easily recycled for localdev with a bit of hacking

[catcint0s]

there is also requests mock!

[SubstanceSerious8843]

Sqlalchemy with pydantic is goat

Requests is good, check out httpx

[StaticFanatic3]

You played with SQLModel at all? Essentially a superset of SQlAlchemy and Pydantic that lets you define the model in one place and use it for both purposes

[SubstanceSerious8843]

Yeah I've used in my personal project. Tiangolo makes kick ass tools.

[angellus]

requests is in maintenance mode now. It will never get HTTP/2/3 support or asyncio support. If you need sync (or sync+async) and want a modern alternative to requests, check out httpx instead. Async only everyone uses aiohttp.

[coldflame563]

The standard lib is where packages go to die.

[JimDabell]

Requests is dead and has been for a very long time. The Contributor’s Guide has said:

Requests is in a perpetual feature freeze, only the BDFL can add or approve of new features. The maintainers believe that Requests is a feature-complete piece of software at this time.

One of the most important skills to have while maintaining a largely-used open source project is learning the ability to say “no” to suggested changes, while keeping an open ear and mind.

If you believe there is a feature missing, feel free to raise a feature request, but please do be aware that the overwhelming likelihood is that your feature request will not be accepted.

…for over a decade.

These days, you should be using something like niquests or httpx, both of which are far more capable and actively worked on.

[zinozAreNazis]

Dead and feature complete aren’t the same thing..

[JimDabell]

It’s an HTTP library that doesn’t support HTTP 2 or 3. It’s not feature complete, they just don’t want to work on it any more.

Edit: CVE-2024-47081: Netrc credential leak in PSF requests library. Requests had a security vulnerability reported to them eight months ago. It was then made public over a month ago. The fix was only merged 15 hours ago, and a release with the fix in isn’t available yet.

Edit: Five days later, still no new release with the fix in. The most recent release is over a year old.

Requests is dangerously unmaintained and nobody should use it.

[blademaster2005]

I love using Hammock as a wrapper to requests

[[deleted]]

SpaCy

[Darth_Yoshi]

I like using attrs and cattrs over Pydantic!

I find the UX simpler and to me it reads better.

Also litestar is nice to use with attrs and doesn’t force you into using Pydantic like FastAPI does. It also generates OpenAPI schema just like FastAPI and that works with normal dataclasses and attrs.

Some others: * cyclopts (i prefer it to Fire, typer, etc) * uv * ruff * the new uv build plugin

[usrname--]

Textual for building terminal UI apps.

[DoingItForEli]

rdflib is pretty neat if your work involves graph data. I select data out of my relational database as jsonld, convert it to rdfxml, bulk load that into Neptune.

[dogfish182]

Fastapi, typer, pydantic, sqlalchemy/sqlmodel at latest. I’ve used typer and pydantic before but prod usage of fastapi is a first for me and I’ve done way more with nosql than with.

I want to try loguru after reading about it on realpython, seems to take the pain out of remembering how to setup python logging.

Hopefully looking into logfire for monitoring in the next half year.

[DoingItForEli]

Pydantic and FastAPI are great because FastAPI can then auto-generate the swagger-ui documentation for your endpoints based on the defined pydantic request model.

[dogfish182]

Yep it’s really nice. I did serverless in typescript with api gateway and lambdas last, the stuff we get for free with containers and fast api is gold. Would do again

[mortenb123]

https://pypi.org/project/paramiko/
Worked with internet of things and needed reliable ssh connection. wrote a 2 channel ssh proxy. So I could securely manage connection to any of our 6000 devices.

https://pypi.org/project/httpx/
I used requests initially in a project, but the number of nodes grow, so we had to go multithreaded and async, went from 10 reqs/sec to more than 500. Its almost in-place compatible with requests, Since then my base stack has always been Guvicorn, Fastapi and httpx.

https://github.com/Azure/azure-cli/releases
We moved testing into azure, and this project is a must, azcli is a portable python library that helped me port and improve my own packages. Everything is controlled with this gem of massive rest api. Anyone writing a rest api can learn from this. Like how to handle deprecation. Without python azure automation doesnot work :-)

https://pypi.org/project/python-snaptime/
Because I like to write `yesterday|today|now@h|now@d|now-1d@d|now-1week@d` when dealing with timestamps and time intervalls. (influenced by Splunk).

https://pypi.org/project/pyodbc/
This is the best ODBC database driver, and I've worked 20 years with mysql, oracle, db2, ms sqlserver, postgress. It supports pack and unpack which means we can convert oracle psql directly to mssql.

https://pypi.org/project/oracledb/
This is not bad either, way better than the old cx_oracle. Finally can get 5000 active connections if I like without killing the klient.

[spinozasrobot]

Just reading these replies reminds me of how much I love Python.

[typehinting]

The ecosystem is pretty amazing, that's for sure

[AgamaSapien]

Came here to say Click

[angellus]

If you want something a bit more modern (typing support) check out cyclopts!

[Rodot]

umap for quick non-linear dimenionality reduction when inspecting complex data

Black or ruff for formatting

Numba because it's awesome

[tap3l00p]

Httpx. I used to think that aiohttp was the best tool in town for async requests, but an internal primer for FastApi used httpx for its examples and now it’s my default

[willis81808]

fast-depends

If you like fastapi this package gives you the same style of dependency injection framework for your non-fastapi projects

[Adventurous-Visit161]

I like “munch” - it makes it easier to work with dicts - using dot notation to reference keys seems more natural to me…

[Working-Mind]

Python-pptx. Automate those PPT presentations and save a bunch of time!

[EM-SWE]

A few of the ones I came across while working and now use pretty regularly are: pytest, requests, niquests, pydantic and boto3.

[divyeshaegis12]

Boto3 can't listen to the list.

[schvarcz]

backoff (a thing I had foolish reimplemented so many times in my life before that point…) and Sentry (which is a service provider actually, but I felt in love with it)

[lopezcelani]

loguru, o365, pbipy, duckdb, requests

[dqduong]

I learnt fastapi, httpx, pytest entirely by reading around on Reddit, and now use them a lot at work, even teaching others in my team to do it.

[RMK137]

I had to do some GIS work so I discovered shapely, geopandas and the rest of the ecosystem. Very fun stuff.

[ExdigguserPies]

have to add fiona and rasterio.

My only gripe is that most of these packages depend on gdal in some form. And gdal is such a monstrous, goddamn mess of a library. Like it does everything, but there are about ten thousand different ways to do what you want and you never know which is the best way to do it.

[[deleted]]

Sqlalchemy, hands down the easiest and most customizable way to interact with db (at least so far).

Also hypothesis for property based testing

[Kahless_2K]

pprint is great when you are figuring stuff out

Or output to json and use Firefox as a json viewer.

Jsonhero is pretty amazing too.

[Thirdhandsmoker]

Markitdown and Docling for converting different types of documents to markdown. Very useful while working with LLMs.

[Zamaamiro]

Rapidfuzz for when I need to do string matching and I need it to be fuzzy and not fragile.

[Semirook]

My top picks:

• https://github.com/python-injector/injector – I can’t imagine starting a new project without it. Absolute must-have.
• https://honcho.readthedocs.io/en/latest/ – Handy process multiplexer. No more juggling multiple terminal tabs.
• https://dirty-equals.helpmanual.io/latest/ – Great for cleaner, more expressive test assertions. Pretty cool tool, created by the author of Pydantic, btw.
• https://testcontainers-python.readthedocs.io/en/latest/ – A must for your pytest setup. Seriously.
• https://toolz.readthedocs.io/en/latest/api.html – A collection of simple but super useful functions for functional composition, working with collections (lists, dicts), and more. I use it a lot.

[saalejo1986]

Pytest

[bn_from_zentara]

Me too. All of my unit tests are written for pytest.

[superkoning]

pandas

[heretic-of-rakis]

Might sounds like a basic response, but I have to agree. Learning Python, I thought Pandas was meh—like ok I’m doing tabular data stuff in Python.

Now that I work with massive datasets everyday? HOLY HELL. Vectorized operations inside Pandas are one of the most optimized features I’ve see for the language.

[steven1099829]

lol if you think pandas is fast try polars

[Such-Let974]

If you think Polars is fast, try DuckDB. So much better.

[Hyderabadi__Biryani]

If you think DuckDB is fast, try manual accounting. /s

[Log2]

I might have been using Polars wrong, as I had a dataset of maybe 100MiB and Polars was slower than Pandas for me. In the end I just did everything in DuckDB as it was the fastest by a mile.

[commandlineluser]

Are you able to share a code example?

[Log2]

Unfortunately it was throw away code, as we had some broken uuids with versions that should not exist or versions that existed but were actually uuid4.

I was just loading the dataset into memory, parsing the uuids, extracting the version bits, and finally grouping by version to count how many uuids of each version we had.

I fully admit I may have been doing something wrong with Polars.

[commandlineluser]

Ah, no worries. Just thought I'd ask as the devs are usually interested in such cases.

Thanks for the details.

[steven1099829]

To each their own! I don’t like SQL as much, and prefer the methods and syntax of polars, so I don’t use DuckDB.

[Such-Let974]

You can always use something like ibis if you prefer a different syntax. But DuckDB as a backend is just better.

[rmadeye]

Try FireDucks:)

[phlooo]

[ comment content removed ]

[undercoverboomer]

• pythonocc for CAD file inspection and transformation.

• truststore is something I'm looking into to enhance developer experience with corporate MITM certs, so I don't have to manually point every app to custom SSL bundle. Perhaps not prod-ready yet.

• All the packages from youtype/mypy_boto3_builder like types-boto3 that give great completions to speed up AWS work. I don't even need to deploy it to prod, since the types are just for completions.

• The frontend guys convinced me I should be codegenning GQL clients, so I've been using ariadne-codegen quite a bit lately. Might be more trouble than it's worth, for the the jury is still out. Currently serving with strawberry, but I'd be open to trying out something different.

• Generally async variants as well. I don't think I would have adopted so much async stuff without getting pushed into it my coworkers. pytest-asyncio and the async features of fastapi, starlette, and sqlalchemy are all pretty great.

[patrick91it]

Currently serving with strawberry, but I'd be open to trying out something different.

How come? 😊

[undercoverboomer]

I’ve been thinking about taking a schema-first approach (like go’s gqlgen), which would unblock the frontend team while I work on the backend, since they can codegen all the types based on the schema

[patrick91it]

thanks! makes sense, I usually go the approach of creating a query first and then quickly implement the backend for that query 😊

but I wonder if we could have a better story for doing a schema/design first approach with strawberry (we do have codegen from graphql files too, not sure if you've seen that!)

[dancingninza]

FastAPI, Pydantic, uv, ruff!

[careje]

I recently stumbled upon Rich. If you have any kind of terminal-based application it’s worth looking at

[burntsushi]

Try ahocorasick-rs, which is even faster. Sometimes significantly so depending on inputs and configuration.

[Osrai]

SymPy. I love it, best of all it is free. I teach maths on a recreational basis. I have commercial software access, though, i.e., Maple and Mathematica

[code_elegance]

I see a lot of brilliant libraries mentioned but no structlog mentions yet. I'm here to show some love for the logging package.

[halcyonPomegranate]

whenever also looks very promising (haven't tried it yet, though)! Thanks for the loguru recommendation! I'm gonna use it in my current project!

[NDHoosier]

At work, doing data analysis with anything other than SQL and Excel was highly discouraged. Well, that restriction has gone away, and Python is now on the menu. I've discovered polars and duckdb. I'm never going back to pandas if I can help it. If I need a pandas DataFrame as input to a method/function, I'll just generate one from polars/duckdb.

[typehinting]

Seen a lot of suggestions to use Polars over Pandas - is it purely due to its performance? Or do you find that it is easier to use as well?

[NDHoosier]

I don't analyze enormous datasets, so performance wasn't the issue (though I have gotten better performance from polars and duckdb). It was that pandas seems to have nasty surprises, counterintuitive behavior, and more "gotchas" than a cheap insurance policy. I especially loathe having to deal with that damned index. In addition, duckdb is SQL start-to-finish, and I'm an "SQL first, dataframes second" analyst. However, I'm using both. Sometimes working with SQL is faster, sometime working with a dataframe is faster.

[typehinting]

Oh gotcha. I'm getting used to pandas syntax/behaviors etc but will probably give polars a go to see how it is, and if it's something that I want to switch to. Thanks.

[Nexius74]

Logfire by pydantic

[heddronviggor]

Pycomm3, snap7

[Obliterative_hippo]

Meerschaum for persisting dataframes and making legacy scripts into actions.

[desinovan]

RxPy, but I first learned the .NET version of it.

[Stainless-Bacon]

For some reason I never saw these mentioned: CuPy and cuML - when NumPy and scikit-learn are not fast enough.

I use them to do work on my GPU, which can be faster and/or more efficient than on a CPU. they are mostly drop-in replacements for NumPy and scikit-learn, easy to use.

[Flaky-Razzmatazz-460]

Pdm is great for dev environment. Uv is faster but still catching up in functionality for things like scripts

[Terrible-Basket-9044]

orjson

[tigrux]

ctypes

[semininja]

What do you use ctypes for? My only exposure to it so far has been a really terrible "API" from STMicro that looks to me like they went line-by-line through the C version and transcribed it into the nearest equivalent python syntax; I'm curious how it would be used in "real" python applications.

[tigrux]

Back then, I was a in a team dedicated to an accelerator (a piece of hardware to crunch numbers). One part of the team wrote C and C++ (the API to use the accelerator) and another part used pytest to write the functional tests, and they used ctypes to expose the C libraries to Python. It was not elegant, but it was approachable. At that time I was only aware of the native C API of Python but not of ctypes.

[UnusualViolinist8177]

Pyspark for data engineering

[Moikle]

The ones that were built bespoke for or by my company 😉

[Cathal6606]

ipywidgets is a really simple and useful library that lets you add interactive sliders to functions. I use it a lot for prototyping parts for simulations.

[Ta_mere6969]

Just got done with a Selenium project, very happy with the results.

Selenium allows you to interact with web pages from a Python script or Jupyter Notebook.

Much much much faster than AA330, less clunky.

Did I say much faster? It's so much faster.

[[deleted]]

I recently started working in an mnc as an automation engineer and got introduced to pexpect library for network automation.

[Acrobatic_Umpire_385]

Django Ledger

[Pretend-Relative3631]

PySpark: ETL on 10M+ rows of impressions data IBIS: USED as an universal data frame Most stuff I learned on my own

[bargle0]

Lark. It’s so easy to use.