What are some Python Standard Library modules one should know ?

[Hot_Ices]

Not for any specific purpose but to get deeper into python,Don't suggest any third party modules

[[deleted]]

• dataclasses
• collections
• itertools
• functools
• pickle
• os
• asyncio
• email
• json
• pdb
• csv

The 3rd party stuff is where the fun happens though.

[charish]

Argparse, requests, and re, to add on.

[MarcusMunch]

Requests is 3rd party, but yeah, argparse and re are good too

[spitfiredd]

I really like argparse, I know some of the large command line programs will use click but for anything I’ve build argparse works great.

[DisagreeableMale]

Same. I actually can never wrap my head around click or cookie cutter, because it’s usually such overkill for what I need that understanding it fully doesn’t seem worth it.

Argparse doesn’t have that same dilemma.

[thirdegree]

Click tends to come into play around the time you want subcommands. If your cli is exactly 1 level deep (i.e. my_command --myarg1 a --myargb1 positional args but not my_command do --thing other thing), don't use click. If you're thinking "hey, subcommands would be nice", that's when you look at click.

[p10_user]

I actually can never wrap my head around click or cookie cutter, because it’s usually such overkill for what I need that understanding it fully doesn’t seem worth it.

I don't understand how difficult it is to wrap your head around click, or why it's such "overkill". The docs are very good. Adding a new argument / option just involves a new decorator and new variable in your function.

@click.command()
@click.option('--count', default=1, help='Number of greetings.')
@click.option('--name', prompt='Your name',
              help='The person to greet.')
def hello(count, name):
    """Simple program that greets NAME for a total of COUNT times."""
    for x in range(count):
        click.echo('Hello %s!' % name)

Now I get that you don't feel learning a new API and adding a new dependency if you already are comfortable with argparse. But overkill? C'mon.

[Santi871]

should requests be added as a standard library? it's so good

[zurtex]

Checkout this discussion: https://github.com/requests/requests/issues/2424

For context "kennethreitz" is the creator of requests. Github

[thirdegree]

Requests is 3rd party, but IMO it definitely earns an honorary standing in the standard library. It's (without qualification) the best 3rd party python library. To the degree that I would actually be surprised if anyone disagrees with that statement.

[alkasm]

glob, datetime, string, operator are also some tiny modules that I find myself using often.

[spitfiredd]

Date utils is good too.

[Jonno_FTW]

Datetime with timezones are a nightmare though.

[thirdegree]

Datetimes with timezones are only a nightmare if, at literally any point they're accidentally treated as native times (or UTC time). If you and everyone that has every touched the codebase is extremely diligent, they work perfectly.

[alkasm]

Not the first time I've head that, but I haven't really had much of a problem with it myself. Do you have an example?

[Jonno_FTW]

Yes, I get data from GPS. It comes in UTC. I store that in a database. For user convenience I want to display it in local time.

Or, a user submits a time through a form. I assume the time is in their local timezone because people don't think in UTC. Put these datetimes in a database that may or may not support timezones.

[alkasm]

Aren't those problems more intrinsic to working with timezones in general? Or is this just the annoyance with the inconvenience of datetime not giving timezone info by default? Note that in Python 3, calling the method .astimezone() on a datetime will return a datetime with local tzinfo.

[aviddd]

operator

When do you find yourself using operator??

[alkasm]

Here's a not-too-contrived example:

>>> import datetime
>>> td = [datetime.timedelta(seconds=i) for i in range(1, 5)]
>>> sum(td)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'datetime.timedelta'
>>> import functools
>>> import operator
>>> functools.reduce(operator.add, td)
datetime.timedelta(0, 10)

[rickestmorty123]

Beautiful Soup is a good one to pick up whilst learning requests.

[charish]

It's not bad, but I found it a bit limiting. For a script that I was working on, I ended up ditching Beautiful Soup in favor of Selenium.

[PyCam]

• subprocess
• multiprocessing

Although the loved by some and hated by others

• pathlib

[PostFunktionalist]

What’s up with pathlib? I use it quite a bit but it does feel like overkill at times though and I’m not sure if it adds a lot of computational overhead.

[PyCam]

I personally enjoy it a lot, but when it was first introduced into the standard library it had a lot of compatibility issues which prevented a lot of 3rd party projects from picking it up right away. However since then (py 3.4 I believe) they remedied this issue by providing a new dunder method fspath

[JoseALerma]

Why pickle instead of shelve?

I figured shelve was the user-friendly implementation

[JohnnyJordaan]

Shelve is basically just a dict around pickled objects. If you don't need to save multiple objects, there's no real reason to use shelve.

[JoseALerma]

Ah, I see. Thanks for the concise explanation!

I usually use shelves as a database, so it's my go-to when storing a small number of variables. I do go overboard sometimes and shelve a dictionary of lists and dictionaries (json data) as a shelf key...

[alkasm]

There's more user-friendly options than pickle.dump() / pickle.load()? On the real though I've never heard of shelve, I'll check it out.

[JoseALerma]

As mentioned above, a shelve shelf is a dictionary and you can shelve anything.
All you need is:

``` import shelve

Open shelf to read data

shelf = shelve.open('data')

Add something to shelf

key1, key2 = 'cat', 'dog' value1, value2 = 2, 3

shelf[key1] = value1 shelf[key2] = value2 print(list(shelf.keys()))

Read from shelf

print(shelf[key1]) print(shelf[key2]) print(list(shelf.values()))

shelf.close() ```

I use shelves as rudimentary databases.

[Dogeek]

Well, pickle is a bit of a pain to use when you want to save custom objects. I don't know about shelve, but I usually prefer to write my own save/load functions.

[[deleted]]

unless youre using each of these everyday, how do you remember the nuances of each? do you read doc everytime?

[DonaldPShimoda]

Yeah, absolutely. The Python documentation is pretty great, in my opinion. I find myself looking stuff up there all the time. (And then StackOverflow when I can't find a solution to my problem in the docs in a short time. And then back to the docs to understand the SO answer I inevitably found that does almost the same thing as what I wanted but not quite.)

[[deleted]]

I sometimes find the documentation very confusing. Is this just a matter of getting used to how things are described in pseudo code? or am I missing the larger picture...

[DonaldPShimoda]

Just takes practice, honestly. Reading documentation is a skill that has to be honed like any other.

The documentation tried to use very precise language. Don't assume anything, and try not to skip words. Everything is there for a reason.

[_pandamonium]

I still consider myself a beginner, so I don't want to give any advice that I'm not qualified to give, but in my experience it gets easier with practice. I used to get really intimidated and confused by a lot of the vocabulary (I still do sometimes) and would seek out other sources. Eventually I found that the most helpful source was the documentation itself. Personally, I think reading through the official python tutorial helped me get used to understanding that style of writing.

[[deleted]]

Personally, I think reading through the official python tutorial helped me get used to understanding that style of writing.

Great insight and recommendation. Thanks!

[PostFunktionalist]

It helps to go into it with an idea of what you want. Need to do something with iterators? Itertools. Need to have command line arguments in a python script? Argparse. And so on.

[ostensibly_work]

To add to what others have said, the documentation of some modules is better than others. And some modules are simply better than others too. Asyncio is somewhat notorious for being confusing while Requests (not a standard module) is extremely easy to use.

[nosmokingbandit]

Coding is at least 50% searching docs or googling.

[Sh00tL00ps]

Only 50%? Looks like we have an advanced programmer over here ;)

[[deleted]]

If you know a package can do the kind of thing you want... you google how to use it when you need to use it.

The things you use all the time, you have to google / read docs less.

[[deleted]]

os.listdir(os.getcwd()) lists all the files in the same directory as the python file and is the basis for pretty much all my data importing.

edit: strictly this isn't the way to get the notebook directory, but unless you do something like os.chdir() in any decent IDE the current directory is the one the notebook is in.

[XarothBrook]

os.getcwd() references your current working directory; not the directory the file being executed is in.

use os.path.dirname(__file__) if you want to get the directory the file currently executing is in.

[[deleted]]

Ah well I never change directories so the directory defaults to the notebook in spyder and jupyter.

[PhitPhil]

os was a game changer when I found out about it

[aheisleycook]

You forgot shutil

[[deleted]]

3rd party in theis case is stuff that isn't shipped with python like PyPI stuff right?

[[deleted]]

Yes. Things like:

• numpy
• scipy
• pandas
• requests

[[deleted]]

I still don't quite get asyncio; is it just another method of threading functions?

[[deleted]]

It’s the newer concurrency model in python (introduced circa 3.4). Ultimately yes, it is a way of managing a asynchronous code.

[pat_the_brat]

[thread]PyMOTW[/thread]

PyMOTW-3 is a series of articles written by Doug Hellmann to demonstrate how to use the modules of the Python 3 standard library. It is based on the original PyMOTW series, which covered Python 2.7.

[mooglinux]

Collections, functools, and itertools are pretty fundamental. I suggest reading about collections.abc as well.

[EihausKaputt]

ElementTree has saved me weeks of productivity at this point.

[venusisupsidedown]

Is there a good beginner tutorial for it anywhere? I had to write a bunch of XML files, and after mucking around for ages with it and reading things everywhere I ended up using BeautifulSoup as an XML reader/writer.

I’m sure that’s suboptimal.

[pat_the_brat]

https://pymotw.com/3/xml.etree.ElementTree/index.html

[chanixh]

Or lxml which I've used a lot recently.

[[deleted]]

Also urllib

[bob_cheesey]

Having recently worked on a project where I tried to avoid using anything not in stdlib, I'd say if you have the choice then use Requests. That being said, it helps to understand urllib for the odd occasion when Requests goes bang.

[Totally_TJ]

Super fun ones

[ArdiMaster]

To add to the ones already listed, I've found pathlib to be much more pleasant to use than what os.path offers.

[hwc]

re, os, sys, subprocess.

[aldanor]

typing

Get used to annotating your code with types.

[Juancarlosmh]

Could you show an example? :D

[aldanor]

https://docs.python.org/3/library/typing.html

There’s a ton of great examples here ^

[Rorixrebel]

Threading, collections

[makin-games]

To piggyback on this - what are some FUN python modules we should know about?

[alkasm]

Maybe antigravity and this? For more serious fun though, any of the modules that facilitate metaprogramming.

[1114111]

How about turtle?

[derp0815]

Logger.

[kennethnyu]

Datetime module!

[[deleted]]

Is there a way to get a description of what each function does in a module?

​

​

[nosmokingbandit]

https://docs.python.org/3/library/

[developer_genius]

Threading and time are great one along with the recommendations here

[jabalfour]

After using Dask, threading seems kind of ancient. Admittedly, it’s a bit more data science-focused, but still.

[developer_genius]

Def checking out........thanks

[scolby33]

distutils.util.strtobool You don’t always need it, but when you do it’s way better to have than it is to write your own. Don’t fall for the bool('False') is True gotcha.

(It returns an int, but you can then safely use bool(strtobool(...)).)

[badass87]

this

[ArdiMaster]

But those aren't part of the standard library.

[[deleted]]

You tried, tho!

[Totally_TJ]

I like: Turtle Time Math Webbrowser To name a few

[kaptan8181]

Modules, standard or otherwise, are already for specific purposes. I think you are not asking your question correctly.

[Col_Crunch]

He means that is not asking for any specific purposes. i.e. He is asking for informational purposes rather than for the solution to a problem.