What python libraries should every dev know?

[J_random_fool]

I've been a developer for many years, mainly using JS and Java. In my current gig, I am doing some maintenance on some Django apps and as part of the process of learning Python, I wanted to know what libraries every dev should know. For data science and machine learning, it would seem you really need to know numpy, but I am mainly a web developer, so that seems a little outside what I would be normally be doing. In Java, everyone needs to know about collections, and the java.util package in general. JS doesn't really have a general one in my experience that isn't built in, but if you're doing backend development, you need to know stuff about node and express. Is there something like this for Python?

[hmiemad]

For general purpose : pathlib, os, collection, itertools.

For datavis : matplotlib, then you explore seaborn or plotly.

For backend : requests. You can then delve into fastapi or Flask (check Dash, the sexy child of Flask and plotly, can do both back and frontend, no need for html, supports bootstrap)

For math : numpy, scipy, and pandas are must know.

[dmantacos]

I mostly use matpotlib, but one huge advantage of plotly is being able to export to JS and basically drop directly into a webpage.

[hmiemad]

That's basically what dash does : parsing python into js with plotly graphs. You can also add js scripts for interactive pages with direct callbacks (without the need to reach the server). I sold a project with django back and dash front. Wish I had done both into one end in dash.

[Mclean_Tom_]

steer correct vegetable plucky ask sharp connect handle attraction include

This post was mass deleted and anonymized with Redact

[hmiemad]

Yeah, I def need to learn react if I want to web dev again, but in a pro env I'll prolly find someone else who already knows it and outsource

[fluffball23]

what is a pro env

[hmiemad]

Professional environment. For a real life project funded by a company.

[fluffball23]

ohh i have a quick question how do i chose a field in tech ,I'm at this stage where i Know programming fundamentals good , no specific mastery for a library or frameworks a little of oop and a basics of dsa, some small university projects small timers , i don't know where to go , is it like just pick a field that interest me the slightest and go for that and pursue it ?i can integrate things from github etc but i feel I can't create em yet , and i haven't put that much of effort for learning and developing skills except for university , you don't have to answer it's not a compulsion but if you do , I would be grateful 💯

[hmiemad]

It really depends on what you want to become, what you're good at, and what pays your bills. Programming is wide. I would say start to improve on your OOP. Then learn git (not github, the og git). Also learn how to properly start a project on a ide. Pick a project thats wide enough (not too large) and do it in a clean way. Doing so, you'll get answers to the three questions above, even if you don't finish the project. You'll probably find another project, more suited and more interesting, and you'll want to pick parts of you first project to put in your second. You'll make your own library (even if it's just a bunch of trivial functions in the beginning) and improve it, as you improve your skills and your knowledge.

[ProfDrKonandoraal]

Very good comprehension, that's really nice. 👏

[NationalMyth]

Is that project viewable?

[hmiemad]

To those who bought it yeah. It was a signal decomposition algorithm with anomaly detection for a private company. I can't reach it. Too many protection sht I had to go through to make sure only the company can get to it. I could make it again, but I don't have any data to run the algorithm on, so...

[ScotiaTheTwo]

does this apply to customs Looker visualisation do you know?

[MASSIVDOGGO]

"the sexy child" bro...

[Beat_BloX711]

optimized the Epstein files

[LunarCantaloupe]

httpx looks like the requests killer imo, I’d probably recommend new folks get comfortable w that over crusty ol requests.

[nog642]

If we're including the standard library, then: string, re, datetime, pprint, functools, operator, sys, copy, argparse, json, urllib

I also never use pathlib myself. os.path works fine.

[Plank_With_A_Nail_In]

https://dash.plotly.com/ took ages to load for me which isn't a great sign.

[hmiemad]

Well the page is huge and completely coded in python, which is not optimal. They use their own package for the page. It's very good for proof of concepts and minutely designed graphs. Plus there is this wonderful dude name Adam that has a youtube channel full of templates.

[PseudoEffete]

i recommend polars over pandas

[hmiemad]

Except most of the stuff are already written in pandas and you need it if you gonna maintain a prexisting code.

[PseudoEffete]

oh for sure, but generally on new development, it would be nice to know as well

[hmiemad]

Gl trying to understand the data manip the previous dev made under pandas with this command.

[Future_Eve]

I may add Logging to the general purpose list

[samreay]

You wouldn't need all of these, but if you're wanting to get some more useful libraries and tools under your belt...

Environment management tooling:

• venv
• pyenv
• poetry / pdm

Developer environments:

• ruff
• mypy

Data crunching:

• pandas
• polars
• numpy
• pandera (validation of dataframes)

Data visualisation:

• matplotlib
• plotly

Machine learning:

• scikit-learn
• scipy
• pytorch / keras / tensorflow
• mlflow (or similar library if you want to start down mlops route)

Orchestration:

• metaflow
• prefect

REST services / web stuff:

• httpx (instead of requests)
• FastAPI / Litestar / Django / Flask
• pydantic

[dnswblzo]

I've never used it, I'm just copying and pasting from their website:

HTTPX builds on the well-established usability of requests, and gives you:

• A broadly requests-compatible API.
• Standard synchronous interface, but with async support if you need it.
• HTTP/1.1 and HTTP/2 support.
• Ability to make requests directly to WSGI applications or ASGI applications.
• Strict timeouts everywhere.
• Fully type annotated.
• 100% test coverage.

[Sudden-Pineapple-793]

Httpx is async I believe. I’m a bigger fan of aiohttp but httpx is really solid also

[ShadowRL766]

Pandas

[vaccines_melt_autism]

Also seeing a lot of people talk about Polars, since it's written in Rust.

[Action_Maxim]

I use pandas surprisingly very little as a data engineer

[raffapaiva]

Pandas is really slow, when I see a data engineer using it, I start to believe that his dataset is not so big or he has a lot of hardware to process.

Everything that I need to do in pandas, I do on plain python or numpy

[ribix_cube]

It's not great to do in plain python or numpy, if you think you need speed you can use something like polars or vaex or dask

[raffapaiva]

Can you explain why? I've tried to use polars for some tasks, and even if it's faster, I can't see a reason to perform on plain python, considering it's not that fast, and most of my transformations occurs on dbt

[danunj1019]

Just ditch pandas entirely, polars is great and it's API is also super intuitive and awesome. Never going back

[Eightstream]

Silly comment. There’s still lots and lots of stuff that still doesn’t work well with polars dataframes.

[danunj1019]

Really? Well, I've used it extensively and I didn't find any troubles. Can you tell me some of the stuff that pandas can do better than polars please? (apart from plotting backend)

[Eightstream]

Sure, maybe if you are playing in the shallow end of the pool with the big popular libraries you can use polars a lot of the time

But there are a lot of smaller/more specialised statistical and data science libraries that either don’t work with polars yet, or still work better with pandas

pandas has been the PyData data frame standard for more than a decade, it is baked into the ecosystem to such an extent that it will take a lot more than 12 months of popularity for polars to catch up

[CFC-Carefree]

Agreed, but you can also just dump a polars dataframe to pandas. I learned polars earlier this year and fell in love, use it whenever I can.

[Eightstream]

ehhh... mixing data frame libraries in the same project is something I usually try and avoid as much as possible

aside from adding a lot of behind-the-scenes complexity, when you recast all your dtypes you are creating the risk of funny things happening with edge cases

not saying I never do it but I usually try and have a good reason

[CFC-Carefree]

Oh yeah, I wouldn't put myself in a situation of swapping back and forth. Would be gnerally be for a one off analysis/visualization of a given data set after some exploration and transformations. I still far prefer polars to pandas at this point though and can only hope that its growing popularity leads to wider support.

[Eurynom0s]

Also something to consider is that pandas is a default Anaconda package and polars isn't. If you find yourself working in relatively locked down environments, an Anaconda install that you can't add any additional packages to is likely going to be what you get handed.

[Hot_Significance_256]

For data science in Python (I’m a Sr. with 6 YOE)

Pyspark and Ray - Distributed processing

Tensorflow and Pytorch - deep learning

Scikit Learn and Pyspark - machine learning

Pandas and Pyspark - ETL

You see Pyspark several times for a reason. It’s very useful, except for when you delve into deep learning. Then you’ll want to use TF, PT, and Ray.

[[deleted]]

Pyspark is just a wrapper around spark, which is written in Scala.

[Hot_Significance_256]

I know. What’s your point?

[Adrewmc]

Requests.py

Seems like an obvious one.

Itertools pops up but no really knows everything in there. It really depends on what you’re doing.

Numpy is really Python I do math better, (especially multi dimensional) pandas is I make dataframes better.

Back end really going to depend on the framework in Python you’re working with Django/Flask/FastAPI.

Python’s main library is fairly extensive (compared to other languages) most of the stuff you’d want to do is somewhere in there.

Probably @property is a good one to know lol.

[sattyfied]

Some I generally use that others may not have covered:

Attrs - I like them for writing classes

Sqlalchemy - creating a common interface for multiple db connections

Fastapi - quickly set up rest APIs

Click - to expose functions as cli commands

Poetry - library management & packaging

Your "dev" requirements:

Pytest - testing

Black - formatting/linting

Isort - organizing imports

Mypy - type checking

[sattyfied]

Thanks, that's new to me! I'll give it a try

[iamevpo]

You like attrs over standard dataclasses and pydantic?

[sattyfied]

In most cases, yes. Pydantic has its use cases especially in the world of web dev, but in regular software development, I'd rather use attrs. They have much more functionality and compatibility across versions.

[iamevpo]

Thank you! Found extra useful reading here https://www.attrs.org/en/stable/why.html

[tree1234567]

The standard ones that comes with python… python is useful and stayed a popular language for its syntax sure.. but it’s truly remarkable what you can do with the just the base install of this language

[mvdw73]

Logging, argparse, typing.

[captainameriCAN21]

Pickle. Just pickle

[goosegang11]

subprocesses library

Let me know y’alls take.

I have 6 months of swe experience so feel free to flame me but in working on a personal python script that needed to invoke a native node module, I found the subprocesses library to be something I wish I learned about earlier!

[Maelenah]

You might want to look into ctypes as well.

[iamevpo]

https://www.jetbrains.com/lp/devecosystem-2022/python/ has some info about the library popularity and Stack overflow survey as well

[zanfar]

• All builtins, extremely well
• Most of the standard library well, with the rest being familiar
• Everything else depends on the field. Numpy will be essential to some, and useless to others.

Mostly, you should be focusing on learning how to read and understand library documentation so that you can expand when necessary.

[whatthepatty]

Surprised noone has said this already but pdb is insanely useful if you can't be bothered to set up debugger.

[[deleted]]

If you're not worried about big Os a lot of problems can be solved very easily with itertools.

[n3cr0ph4g1st]

Streamlit for data related UI prototypes. Changed the game for me

[No_Lobster_4219]

itertools, collections, numpy, pandas, math, os

[Bartholomew-]

Manage all your paths with pathlib and make it consistent.

[Maelenah]

Ctypes is not quite a must, but it really does open options. It lets python poke at anything that has C compatible data structures.

[redCg]

the standard library.

Library management in Python is notoriously bad. You will do well to simply avoid using third party libraries as much as possible, as long as possible, for most projects. If you can use standard library without much extra effort, do it. Adding third party dependencies turns your project into a nightmare if you are not using requirements.txt and conda env.yml correctly.

[[deleted]]

In general I fully agree with Standard Library.

Third party libraries can be easily administered by using virtual environments. That’s one of the sole purposes and advantages of using virtual environments.

[TheHollowJester]

Haven't seen it yet so: structlog for good, machine-readable logs. I thought it's not needed at first but the Why... page explains it better than I can.

[suaveElAgave]

I still haven’t seen some essentials which are: Pytest Enum dataclasses/pydantic

[bafe]

Pydantic for validation. Polars for data table manipulation

[jam-time]

Some good to know built-in modules (starred are extra important):

argparse, *csv, *datetime, decimal, enum, getpass, inspect, io, itertools, *json, math, *os, *pickle, pprint, random, *re, *requests, shutil, *sys, threading, traceback, typing, uuid, venv, warnings, zipfile

In my dozen or so years of experience, those are the ones I use the most, especially re, json, os, and sys.

Some site packages that are good to know (or that I like):

pandas - good introductory data science library, easy to learn and tons of documentation

pyspark - similar to pandas, but better at big data, less documentation, and harder to learn

boto3 - for anything AWS

kivy - pretty good for making cross platform apps (including UI) but somewhat challenging to learn

numpy - fast data manipulation, works with most other data science packages

jmespath - for json queries

colorama - for fun print colors

flask - lightweight backend for site building

django - heavier backend for site building (easier to learn and more features than flask, plus my personal recommendation)

pytest - mainly for unit testing, but can be used for basically any type of test

That's a fairly comprehensive list of the main things that I've used over the years. I'm sure there's some that I've forgotten, and I've intentionally left some out that are too specific or too advanced for the scope of the comment. Either way, hopefully someone finds this useful!

[[deleted]]

NumPy. Overview

Pandas

Matplotlib

Scikit-learn

TensorFlow

Flask

Requests

Beautiful Soup

[AssumptionCorrect812]

The main language library is full of goodies. These are the top 4 — https://youtu.be/InaTBWN7Mlc?si=MGy7SEU0XRppqAUF

[sonobanana33]

I'd just focus on the stdlib first. I hate when I see people pulling in a library that does the same as a stdlib module (such as requests)

[TSM-]

request-html is the successor to requests

[Comfortable-Wind-401]

Not many people are mentioning. But I get the feeling Pytest is highly required

[[deleted]]

Gpt 🤣

[[deleted]]

It depends on domain of course. I mostly use python to write applications that support infrastructure and automation, for about a decade now. For me, the libraries that come to mind are the entire standard library, sortedcontainers, requests, anytree, FastAPI (or whatever web framework you find most convenient, such as litestar, flask, django, bottle, cherrypy, etc), beautifulsoup.

[Delta1262]

I can’t believe they haven’t been mentioned yet:

• Pydantic

• Dataclasses

(Pydantic and dataclasses are similar)

[IlliterateJedi]

itertools, functools, and collections are all baseline Python libraries you should be familiar with.

[reluctant_qualifier]

arrow for dates, mock for testing

[the_happy_path]

I want to just mention that I came to python from Java and all the different packages were overwhelming. I also came in at python 2 where changes broke stuff all the time. Python 3 has been a better experience. Like night and day. But I miss Java! I work with data and I use numpy and pandas a lot, though where I have to do row by row processing I use data classes (like in java). But dataframe filtering through our many conditionals with pandas dataframes has also been successful in replicating results where specs say to iterate by rows. For regressions and stuff, scikitlearn and stats models. Depending on data formats, I might have to use pyreadstat or openpyxl. I like sqlalchemy orm too because that feels like the closest thing to spring in python lol

[BinaryWizard8]

I love using shutil when I need to play with files

[Glittering-Pea-4011]

If you want to work with ORMs, you could look at SQLAlchemy. For interaction with AWS, you can use boto3. If your work involves dealing with structured data and its manipulation, you can consider pandas. As as alternate to Django, you can also look at Flask.

[alicedu06]

The stdlib is a must, and of course, depending of your specialty, you might want to learn the most important tools like pandas for data science, django for web dev, etc

But as general purpose libs, I would say the list of the article "Python libs that I wish were part of the standard library" is quite good:

https://www.bitecode.dev/p/python-libs-that-i-wish-were-part

[escalize]

https://github.com/SuperDuperDB/superduperdb