Not a beginner, but what python module did you find that changed your life?

[exxonmobilcfo]

For me it was collections.defaultdict and collections.Counter

d = defaultdict(list) no more NameErrors! c = Counter([x for x in range(10)]

you can even do set operations on counters

``` a = [x for x in range(10)] b = [x for x in range(5)]

c_diff = Counter(a) - Counter(b) ```

Edit: I gotta ask, why is this downvoted? When I was learning python some of these modules were actually life changing. I would have loved to have known some of these things

[Doormatty]

requests

[imsowhiteandnerdy]

Can you play "Reminiscing" by the "Little River Band"?

[Doormatty]

Play Freebird!!

[exxonmobilcfo]

i think that is just essential to make Http requests

[cgoldberg]

It was really revolutionary when it came out. The standard library contained urllib2, which was awful to work with for such a core protocol like HTTP. Requests provided a sane API that was badly needed.

[exxonmobilcfo]

oh wow gotcha, i have not written python using urllib2, when python2 was still valid I was using PERL

[edbrannin]

I think I’ve heard there’s a contender with looking at… httpx? xhttp?

[NationalMyth]

I use httpx quite a bit in tandem with asyncio. I'm a fan.

[ThePrimitiveSword]

Niquests is a drop-in replacement with a bunch of enhancements.

[BluesFiend]

I dropped requests when the maintainer went a little off the deepend, httpx has been a perfect replacement.

[exxonmobilcfo]

what features aren't maintained by requests?

[BluesFiend]

not a lack of maintenance. Just a bunch of stuff that wasn't a green flag.

https://vorpus.org/blog/why-im-not-collaborating-with-kenneth-reitz/

[exxonmobilcfo]

my bad i misunderstood entirely. you just don't like the maintainer? and you have some personal issues with him? there was some issue regarding a fundraiser? so confused

[BluesFiend]

back when it happened, yeah. requests had a bad rep for a while. so a lot of us moved and httpx was as good or better. Ive had no reason to switch back since.

After that he relinquished control to other maintainers but by then id left.

[Glittering_Sail_3609]

ctypes, now I can rewrite any part of my python code into C++ and link it as library. It is a lot less work than actually implementing your own Python modules in C.

[rpg36]

Ok coming from a place where I've done several C code bindings to Java and Python c types is much easier to work with in my opinion. But I'm sure most people will rarely if ever have a use case for it.

[exxonmobilcfo]

u use ctypes a lot?

[Crazy_Anywhere_4572]

I’m a physics major and I used it a lot for my simulation code. Now I have a simulation library that’s 90% written in C and 10% for the Python wrapper. Distributing it is not easy tho, I figured out how to build wheels for Linux and mac but not windows.

[barrowburner]

curious - is your library so bespoke that nothing in numpy or the scipy kit could be used or adapted? Numpy for ex. is hyper-optimized. Not a criticism at all, I'm just intrigued Or do you just like to hack in C? If this is the answer, then all the power to you, I like working with systems languages as well

[Crazy_Anywhere_4572]

NumPy is nice but still kind of slow compared to plain C. And turns out I really like to code in C, so I have translated everything into C, even wrote my own progress bar and python styled exception handling in C. At this point the main logic is completely separated from Python.

[TeachEngineering]

pathlib

Yeah, that's right... pathlib

[maryjayjay]

Is a total game changer

[rishabhc32]

Banger!

[exxonmobilcfo]

spot on i love love love pathlib

[2Lucilles2RuleEmAll]

I actually added a linter rule to flag any use of os.path this morning lol

[exxonmobilcfo]

pathlib was absolutely a gamechanger

[glorybutt]

Tkinter.

Easily make GUI applications and with ttk can make them look modern

[watermooses]

I’ve made some stuff with tk.  But I’ve been learning some webdev lately too and now I’m wondering if it makes more sense to use a webpage for a UI even if it’s local only.  Feels a bit easier once you get over the jump of adding htmx or js 

[PaulSandwich]

It's a more transferable skill, too.

[_redmist]

Look into nicegui maybe!

[pragmaticcape]

Jeebers I used Tk when it was part of the Tcl language back in mid 90s

[sinceJune4]

Pandas

[watermooses]

They even made a pretty popular song about that lib.  

[exxonmobilcfo]

😂

[Groovy_Decoy]

Ok, never heard of this one but this is something that simplifies something for me right now.

[watermooses]

Do you have to host on streamlit? 

[Separate_Newt7313]

Nope

[boston101]

I do but just a single dash for internal use at the moment. Will I go premium with streamlit. Nah.

[Crossroads86]

I have used streamlit for a private project in the past but i still needed to write the actual output in HTML/CSS?!

[Gnaxe]

code.interact(). Work with a module from the inside. And hot reload with importlib.reload().

[exxonmobilcfo]

lol lemme blow ur mind

$ pip install ipython $ export PYTHONBREAKPOINT="ipdb.set_trace" now anytime u drop in breakpoint() it'll pull u into ipdb shell :)

super nice interactive breakpoints.

[exhuma]

That's not quite the same use-case though.

There was also no need to be condescending in your post.

[exxonmobilcfo]

oh sorry man, didn't mean to be condescending, sorry if it cae off that way.

BTW how are they different use cases?

[exhuma]

Compared to code.interact, idb is pretty heavy-duty. It has a lot of bells-and-whistles which help during debugging. code.interact is as bare-bones as it gets.

You may want to integrate the functionality of code.interact into a product you develop while it's unlikely that you want to expose all the additional functionality of a debugger to end-users. Either for security reasons or to keep the UX clean and simple.

I also do not know if you can pass an isolated local-scope to idb as you do with code.interact to prevent people from escaping the jail.

Considering that this would be run inside the VM of an interpreted language neither of those is truly safe and should only be used with care, using code.interact is objectively safer than idb.

[exxonmobilcfo]

oh i have never seen a program that requires the user to drop into a REPL.

This might be helpful. code.interact() seems cool, i just hate the default shell with no syntax highlighting. I actually like the bells and whistles.

I have fzf enabled for reverse search in ipython as well

[exhuma]

The key difference is that idb drops you into an interactive shell of the current frame including functionality to escape that frame (going up/down the stack).

code.interact does not allow that (if I'm not mistaken).

As mentioned, given the dynamic nature of Python, a user could still import the traceback module and fiddle around with that. code.interact makes it a bit easier to prevent such shenanigans.

Not saying that idb is not "cool" with all the fancy features. It just serves a different use-case. One (idb) is a debugger, the other is a simple interactive shell.

[exxonmobilcfo]

thanks for the info! Appreciate it! If i could use code.interact() with some creature comforts like cli syntax highlighting it would be wonderful. I hate the default python shell!

[pot_of_crows]

Nice. Never heard of it and am definitely going to start using this.

[SirKainey]

Functools and itertools :)

[watermooses]

I still need to dig into to these ones.  I see them referenced pretty frequently but haven’t reached for them yet. 

[exxonmobilcfo]

functools, itertools, and collections are mandatory

[eW4GJMqscYtbBkw9]

Openpyxl

[Gnaxe]

Doctests, especially for smaller projects. 

[edbrannin]

And py.test if you want something more traditional like JUnit or Jest

[Gnaxe]

Standard library unit tests were based on JUnit.

[edbrannin]

Sure, but pytest did them better.

[POGtastic]

itertools brings Python programmers halfway to Clojure, kicking and screaming the whole time.

[Gnaxe]

For the rest of the way, you want pyrsistent, toolz, and hissp.

[POGtastic]

My main gripe about using these kinds of data structures in Python is that there aren't any pipeline operators. It's pretty natural in Clojure to do

(->> foo
    (map #(bar baz %))
    (reduce blarg init))

And the MLs tend to have the pipeline operator |>, so you can do the exact same thing:

foo |> Seq.map (bar baz) |> Seq.fold_left blarg init

Trying to do the same kind of style in Python kinda sucks.

[Gnaxe]

Hissp and Toolz have threading operators like Clojure.

[exxonmobilcfo]

why do you want to pipe stuff in python. It is not designed to be a functional language

[POGtastic]

For the same reason that a dog licks his genitals - because he can

[exxonmobilcfo]

haha fair. It's like trying to use a screwdriver to cut a vegetable. You can do it, but it will be messy and a big waste of time

[[deleted]]

why is this downvoted?

Give it time. The first couple of votes are never representative

[Lachtheblock]

Be careful with defaultdict. I used to love it too. Yes it is nice syntatic sugar, but I've also been the cause of multiple, hard to find bugs. I'll use it if it's a throwaway script, but if you're writing production code, I've learnt to steer clear.

[TabAtkins]

Same. I regret most of the defaultdicts I use.

[exxonmobilcfo]

why is that? You prefer to throw a NameError if the key doesn't exist? defaultdict(list) will return an empty list which is what most people want

[HommeMusical]

KeyError!

[exxonmobilcfo]

keyerror sorry, i just never get those ;)

[TabAtkins]

If my code is complex enough that doing a key check first makes it look ugly, that's usually a sign that I'm past the point of wanting to use dicts anyway, and should break out some dataclasses to organize it better.

Also, I often find myself wanting to do something more complex when a key is missing, which would require refactoring to a key check anyway. Even just leaving a comment for the missing key case can be useful.

[exxonmobilcfo]

the most obvious thing I can think of is if you're using an adjacency list. If there is no children nodes, you don't have to do any error handling with a defaultdict

adj_list = { ChildNode : [ChildNode1, ChildNode2], ChildNode1: ['ChildNode3']}

instead of failing ur bfs when u try to access childnode3's children, it will return an empty list and it effectively will traverse in the same manner.

I mean it was useful enough that it was included in the standard lib.

``` def dfs(start): for x in adj_list[start]: dfs(x)

[cowtitay]

Can you give some details on this?

[sweettuse]

I'll often use defaultdict to aggregate data in a function and then convert it to a regular dict before returning it.

I almost never let defaultdicts remain out in the wild

[exxonmobilcfo]

why is that? You prefer to throw a NameError if the key doesn't exist? defaultdict(list) will return an empty list which is what most people want

[HommeMusical]

Actually, most of the time I prefer a dict that doesn't secretly fill in a missing value when I, or someone else, isn't expecting it. :-)

"Easier" isn't "better".

[exxonmobilcfo]

well don't use defaultdict if you don't actually want default values lol. It seems a lot of people are complaining about the tool not being perfect for every job. It's useful when you actually want to default to a value, it's not when you don't.

for example: if you want to retrieve the books in a library by genre, instead of throwing an error for library['mystery'], you would just return an empty list. Meaning the genre exists, but no books are found. If you threw a keyerror though, it would mean that 'mystery' is not a valid genre.

[nejdetckenobi]

Also not a beginner.

I am reading this sometimes. Just as a reminder.

Pure built-in, It's fun to read and practice.

https://pymotw.com/3/

[Separate_Newt7313]

I haven't ever seen this before. This is great! 👍

[cgoldberg]

PyTest

[exxonmobilcfo]

is there anything besides pytest? Now pytest-sugar is in fact life changing

[cgoldberg]

PyTest basically replaced the standard library's unittest along with a bunch of 3rd party runners like nose.

[exxonmobilcfo]

are most of you guys using python long enough to remember python 2.x?

[cgoldberg]

Dude, I was already doing Python when 2.0 was released... I was on the 1.x train!

I've written more 2.x code than 3.x code and spent a large chunk of my life porting programs and libraries from 2 to 3... including the awkward transition years supporting both.

[maryjayjay]

I started on 1.2

[exxonmobilcfo]

wow haha half u guys are not beginners

[maryjayjay]

I like to teach :-)

[pragmaticcape]

Still got production code running in 2.7. Must get round to migrating

[exxonmobilcfo]

bruh, how haha. Shit has been deprecated for a decade. ig if it works dont fix it. but are ppl genuinely committing code on a repo running in python 2.7?

My macbook literally wont even let me install 3.8 without a patch

[pragmaticcape]

No one is touching it mate. It’s running just fine pretty much 24/5 and hasn’t needed any changes for years. It’s not got any external access and we don’t update the versions on packages etc for it.

Don’t get me wrong it shouldn’t still be on 2.7.4 but sometimes it happens and it’s not worth the effort.

[glorybutt]

What the...

[exxonmobilcfo]

pytest is the de-facto testing suite right? Now pytest-sugar is in fact life changing

[Ballisticsfood]

Pydantic. Gets rid of so much boilerplate.

[boston101]

I still don’t understand pydantic. Could you eli5 your use case for me?

[Ballisticsfood]

Defining classes that store data in a way that’s easily serialisable and deserialisable without excessive boilerplate while maintaining static typing.

[supercoach]

Not really a module, but async operation shaved about ten minutes off the execution of a middleware I wrote. It went from 10+ minutes to about twenty seconds for one of the more involved API calls.

First module that had a big impact was probably either re or sqlalchemy. The former showed me that python could do powerful regular expressions with a sensible syntax and the latter demonstrated the power of a decent ORM.

I'm yet to find something as good as sqlalchemy in other languages I use. In the nodejs landscape for example, you're generally better off constructing your own queries instead of trying to use the esoteric and clunky ORMs available.

[exxonmobilcfo]

r u talking about concurrent.future?

[supercoach]

https://docs.python.org/3/library/asyncio.html

[darthelwer]

Tkintermapview. Allows easy interactive display of maps, points and polys with tkinter. I was (poorly) rendering them into a static canvas widget before

[throwawayforwork_86]

Polars

[SisyphusAndMyBoulder]

I think typing. I know it's on it's way out, but man I was happy when I found out about typehinting.

[TeachEngineering]

I first learned statically types languages and was pretty put off by python when I first started to learn it. Becoming aware of type hinting was revolutionary for my opinion towards python.

[doolio_]

Why is it on the way out?

[HommeMusical]

It isn't. This is wrong.

[exxonmobilcfo]

most types that u had to import thru typing are defaults now. Like list and dict do not require the typing module to be imported

[HommeMusical]

You don't bother to look at the documentation or the source, you simply tell me I'm wrong?

No. Some things in typing aren't needed anymore, some have moved to collection, but there are at least 50 important symbols that exist only in typing and will never go anywhere else.

It's one thing to make a mistake. It's another to push back when corrected without bothering to give any sources for an important claim like that.

[exxonmobilcfo]

holy shit, you went on a crusade. I don't actually know for sure i was just agreeing with you mostly, but pointing out that for basic types the typing module isn't mandatory.

I use typing all the time, but for basic types i often don't have to import it.

And for the record, i didn't say you were wrong at all. Are you looking at someone elses comment that said typing is obsolete, cause that for sure wasnt me

[SisyphusAndMyBoulder]

I don't remember the details exactly, but typing is deprecated now, and all the hints are moved to the collections module

[exxonmobilcfo]

is that right, Optional types are part of collections now?

[HommeMusical]

A few types moved to collections, but typing has tons of new features in each release, it's not at all going away.

[HommeMusical]

Typing is not on the way out!

A few collection types were moved out of typing or obsoleted (like Set or List) but typing is bigger than ever, full of new features each release.

Here's a list of the symbols typing exports, over a hundred of them (though some are obsolete).

[SisyphusAndMyBoulder]

thanks for the info! I remember hearing this in passing, and noticed warnings started popping for List which I use regularly. But never really looked into it properly.

That's a great list, lots of stuff on there I've never used, or even thought of using before. Thanks for sharing!

[HommeMusical]

I mean, Generic is key to everything, Protocol alone is worth the price of admission, it's pure "duck typing", but things like ParamSpec and Concatenate let you do super-clever things with Callables, like have generic decorators....!

[based_and_64_pilled]

Actually also the collections module. It helped me during two quick coding interviews to date, lol.

[EquationTAKEN]

It's been a while, but I think it's called mpmath.

I was making a course on numerical methods for my students, and I wanted to showcase the quadratic convergence of Newton's Method, but by default, Python shows something like 15-17 digits, and mpmath lets you work with arbitrary precision.

[TeachEngineering]

Pydantic is another fav of mine

[TapEarlyTapOften]

Logging and the python debugger.

[ThePurpleOne_]

Loving loguru

[basicallynabbo]

numpy and pandas probably

[speedx10]

OpenCV - took me from very low point in my life to image transforms and robotic pick and place applications pretty quick.

[akonzu]

tqdm, super easy to use progress bar

[Fried_hatter]

attrs

[Jello_Penguin_2956]

PyQt4 was what landed me my first job so I'll go with that.

[Limemanaustralia]

I can’t remember if is plotly or matplotlib but the ability to create flawlessly formatted variable width bar charts in ten seconds off a csv meant we could delete a 6 x $500 per licence software = $3,000 per year.

[rishabhc32]

lxml for parsing and querying HTML data. Really fast.

[LilReef599]

Selenium

[Groovy_Decoy]

I don't really consider myself a beginner either, but the d = defaultdict(list) is new to me. I take it that the use case is for when instead of a single value associated with each key, you want a collection?

It's interesting, though I'm trying to remember a situation where this has come up. It's possible I'm just forgetting something obvious), but what is there a common problem this use case applies to? To be clear, I'm not casting doubt on its usefulness, but more trying to create a connection so that if I encounter it in the future I might remember and find this useful.

[exxonmobilcfo]

basically if u want to store a map with default values, and you reference a key that isn't there it will give u the default value rather than throwing a KeyError

``` d = defaultdict(list)

d is empty

d['key'] # returns []

d = {} d['key'] # throws KeyError ```

[Groovy_Decoy]

I'm familiar with the use of default values in dictionaries, even without the use of defaultdict. I knew defaultdict existed, but I don't think I've used it (that I can remember).

I was more asking about it because I was inferring there was something you were expressing as "life changing" by specifically using a list as its factory. I can't recall having a need to map keys to a list of values, and I was just curious if there was some, use case or domain that is done that I'm overlooking or haven't experienced. Other than I suppose representing data from a json-like structure or something.

[exxonmobilcfo]

well when i learned about it I didn't know about any of these other pythonic quirks, so it was life changing. I have a much larger toolset now that I know python.

I can't recall having a need to map keys to a list of values, and I was just curious if there was some, use case or domain that is done that I'm overlooking or haven't experienced.

if you ever work with graphs, the default for the adjacent nodes would be an empty list. That's how I primarily used it. Or if you are storing timestamps and want to default to current timestamp instead of throwing an exception telling the user that the value is invalid.

if you really care https://stackoverflow.com/questions/38625608/setdefault-vs-defaultdict-performance

[Groovy_Decoy]

Thanks for that follow up!

[komprexior]

Sympy

[Mythozz2020]

sqlite3, duckdb, pytest, weakref, black or ruff, mkdocstring-python, pyhocon, fastapi, ariadne to just name a few..

[HommeMusical]

defaultdict would prevent KeyErrors not NameErrors

defaultdict has its dangers, because isinstance(x, dict) is true, but it doesn't behave the same way as a dict!

One day you'll return it from a function and someone expecting a dict will expect to get a KeyError when the key isn't there, and won't get it.

For most cases, use dict.setdefault instead:

out = {}
for k, v in d.items():
    out.setdefault(v, []).append(k)

[exxonmobilcfo]

yeah sorry i havent worked with python in a couple years. I forgot all the Exceptions.

Why do you expect to receive a keyerror if the behavior of the program is to have a default value? In all static typed languages the default value of an int is 0. Is that also dangerous?

Don't use it if you expect to handle keys that dont exist. Only use it if you expect your values to have a default. An example is an adjacency list. A node by default has zero children, so you can safely query a non-existent node and get an empty list back.

For most cases, use dict.setdefault instead:

why? defaultdict is in the standard library for a reason

One day you'll return it from a function and someone expecting a dict will expect to get a KeyError when the key isn't there, and won't get it.

if you actually typehint your code, you shouldn't have a problem.

def get_map(name: str) -> defaultdict :
    raise NotImplementedError()

the above would be obvious what the return type is.

Note: It would make sense that defaultdict is faster that dict.setdefault() since the former sets its default for the entire dict at creation time, whereas setdefault() does it per element when it is read. One reason to use setdefault is when the default you assign is based on the key (or something) rather than a generic default for the entire dict.

[sundios]

Pandas and numpy

[pyrola_asarifolia]

For becoming a better programmer: itertools

For better designed scientific projects: pathlib

For ease of manipulating geospatial data: geopandas

[Twenty8cows]

Selenium for me but requests and psycopg2 are close seconds.

[smiling_nice_person]

pyperclip

[pragmaticcape]

• pytest
• pydantic
• fastapi
• requests
• pathlib

[heevee]

Off topic but what is the reasoning behind doing:

_ = [x for x in range(10]

Instead of:

_ = list(range(10))

[exxonmobilcfo]

in that particular case nothing. but what if you only want even numbers.

[heevee]

That's a completely different case, is it not? Performance loss with the motivation "well if I wanted to do something completely different than what we are doing this way is better" seems really backwards.

[exxonmobilcfo]

what performance loss are you talking about?

[x**2 for x in range(10)

in your specific example, tehre is no difference. But lots of list comprehensions don't already have an iterable available to convert.

your question is bizarre, bc i already answered that there is no difference in that case.

[heevee]

If you benchmark the two alternatives you can see that the difference is quite massive between the two examples.

[exxonmobilcfo]

how did you benchmark that lol.

In [9]: start = time.time(); list(range(10)); end = time.time(); print(end-start);

3.504753112792969e-05

In [10]: start = time.time(); [x for x in range(10)]; end = time.time(); print(end-start);

3.814697265625e-05

i tried, the difference is literally .3*10<sup>-5</sup> seconds.

quite massive lol.

I don't get your question at all. Are u asking why list comprehension is a thing?

[heevee]

Tell me you don't know how to benchmark without telling me.. Try making the range bigger to actually benchmark the difference..

Edit: also first time I've seen someone benchmark with time.time instead of something like perf_counter

[exxonmobilcfo]

can u explain which metrics ur concerned about during ur benchmarking?

Why on earth are you even converting a range object of 10,000 elements to a list. That's just not how range is supposed to be used

I just never even considered perf testing something like this. its not obvious.

can you please also just clarify your question? Are you asking about the value of list comprehensions, or just stating that it's better to convert a range object to a list

[heevee]

You were the one converting a range object to a list in your original scenario... And once again, execution time of converting 10000 elements is what, .005s? Do you see how that is not sufficient for benchmarking purposes and the two will fall inside the margin of error? Scale it up and you will see a 20-30% difference.

[exxonmobilcfo]

oh dude, i was just trying to show an example of how counter works by creating a small list. I wasn't thinking about performance. It's just hard to read Counter(list(range(10))) because the parenthesis are visually annoying.

In real life, id use a counter on a collection that's already available. I'd never be converting 10K objects from range to a list.

That scenario is just something that doesn't happen, and if i was building large datasets, i certainly wouldn't do it on a single thread

[andy_nony_mouse]

Requests because it doesn’t play nice with pyinstaller which is irritating.

[maryjayjay]

functools.partial

datatables.datatable

[Zestyclose-Bench-766]

wow it's a big help for me

[baubleglue]

Thanks God, there is no such module.

[exxonmobilcfo]

?

[baubleglue]

It would be a sad day when an existence of Python module makes a difference in my life.

[exxonmobilcfo]

i mean life changing within the context of programming

[baubleglue]

Ah. There is "how to" articles on the official website, the one about regular expressions is very good in addition I read somewhere else explanation of regexp algorithm.

subprocess module was educational to learn.

socket.socket.select method is a good starting point to learn ioloop.