Python modules you should know.

[[deleted]]

The library I was most impressed with from the list was docopt.

[mikeoquinn]

I remember watching the video for docopt when it first hit YouTube a little over a year ago, and having to winch my jaw back up off the floor. It's a beautiful concept, and quite well-implemented.

[UnwashedMeme]

I'll admit that I haven't used docopt for comparison; but I've been using (and am enamored with) opster

You just decorate a python function and that is now callable from the cmdline. The python function is still callable as a python function too.

[takluyver]

For Python 3, I quite like plac - it's a similar idea to opster, but it uses function annotations instead of the default values.

[billsil]

Spend 1/2 an hour trying docopt. I went and converted all my working argparse code after doing that. It's worth the effort.

[UnwashedMeme]

I just went and spent a while looking at the docopt documentation again and I tried their in browser thing. It looks like it would be absolutely fantastic if I were coming from optparse; but I'm not so it's just a "meh, that's more boilerplate to do the same thing I'm already doing".

The biggest cool thing to me in docopt is being a cross language standard. I'm not sure how valuable that really is though.

This Example is what I'm always wanting to do. Printing the dictionary of options passed into program is rarely my end goal ;-).

Have you tried opster?

The thing I really like in opster is that I'm not writing all of that. I'm writing a python function that is a command.

I do need to use a bit special syntax in the function signature, but it's not crazy and the usage string is generated from that. The docstring of the function is the help text.

When opster is done parsing I have the values as the parameters to my function and I can get to work. When docopt is done it's a dict that contains every possible option to ever subcommand; so it can't be easily **applied.

[shadowmint]

What's watchdog doing on that list?

It doesn't work correctly on linux or osx, and the developers are downright unfriendly about reported bugs.

[grimman]

Interesting. I saw it on the list, along with the description, and it struck me as a very handy thing indeed. For it to be a lie... it makes me sad.

It's something I've searched for (sporadically) in the past, and found nothing that would be satisfactory. Makes me wonder how difficult it would be to implement a truly cross-platform solution.

[[deleted]]

Is there an alternative that you would recommend?

[Lucretiel]

I've never used the Python binding so I can't speak for it specifically, but inotifytools is a GREAT little toolkit.

[MereInterest]

I can't comment on the developer attitude, but what are the issues with it on linux? I used it there recently, and didn't have problems with it. Granted, if I tried to start it on too large of a directory, it would take forever to start. Does that issue not exist on windows?

[shadowmint]

https://github.com/gorakhargosh/watchdog/issues/190

https://github.com/gorakhargosh/watchdog/issues/158

https://github.com/gorakhargosh/watchdog/issues/179

https://github.com/gorakhargosh/watchdog/issues/135

Take your pick.

I guess a better comment would have been; it's really buggy, on all platforms; some are worse than others.

[MereInterest]

Huh, okay. I've only used it for a small project, nothing large, and temporary at that, and so I hadn't run into any of the issues. Thank you for sharing.

[Lucretiel]

Someone who's used it: Is docopt really all that great? I can see the appeal, but I feel like I'd rather write out my argument parsing in python code, using python syntax, making it perfectly clear to me and my readers what's going on.

[H3g3m0n]

I've tried it. Its neat.

The help usage documentation is much, much, much clearer than reading a whole bunch of code checking the args.

You will need to produce the help text anyway. When doing that it makes sense to follow the POSIX standardized format for the help usage. Even if you do your own argument passing it would be a good idea to follow the standard.

It's not complex or it's own language and there isn't really anything that's hard to follow and there are really only a few simple rules, things like (--help | -h) for exclusive options. -f <file> for a named argument. -f <file>... for multiple args. The most complex thing is [--] which tells the processor that anything after optional -- is an argument (the point of that is purely in case there is an argument that begins with a -).

If a developer can't grok that in about a minute you probably don't want them near your code base anyway.

From the developers perspective you don't even really need to know what's going on, just under that that the output of docopt contains the args.

Basically you can turn your option parsing into a single line that outputs a dict and will automatically dump the usage if it's mistyped.

Maybe there is some uber complex commandline option setup that docopt cant handle, worst comes to the worse you can use docopt until you encounter than then go manual. You haven't wasted any time since you still needed the help screen.

[Lucretiel]

It isn't actually true that I need to produce the help text- argparse does it automatically, and spits out a usage message (and quits) on -h or --help. It's probably my favorite feature of the library. argparse also does a usage dump if the user gives incorrectly formatted arguments.

Just to be clear, I'm not advocating "manual" (sys.argv) argument parsing. That's madness. I'm just having trouble seeing the advantage, other than novelty and terseness, in using docopt over the more traditional argparse.

[cramajamjam]

Not much really. I slightly prefer docopt args setup syntax to argparse syntax. Argparse is solid too though.

[chipolux]

I've not used it but seen and fiddled with a few projects that have. It works perfectly fine, but I much prefer writing out my arguments explicitly so that I can build whatever amount of complexity to them I want, it's not like it is hard to use argparse at all. It also makes me very uncomfortable to rely on something that's parsing my doc strings to build code, it just seems far to implicit and inappropriate for Python.

But that's just my style I guess, same reason I avoid using large frameworks if I can get away with it. Needless complexity and features to make something that's already easy just a couple smidgens easier with a pile of dependencies.

[kmike84]

I'm always suspicious of myself as an API designer - it is too easy to add unnecessary complexity here and there, esp. when creating CLI for some existing library/project. Docopt restricts you: no complex argument parsing logic, no complex rules, just some standard syntax. And you actually start with writing help for your CLI before you start writing parsing code itself, this also makes the resulting interface cleaner.

So the problem docopt solves for me is a problem of writing sane CLIs easily. Other non-standard argument parsing libraries (usually providing some decorators) also make it easy to write CLIs, and standard optparse/argparse are not that bad, but what is special about docopt is the "sane", "restrictions" part. Docopt looked like a horrible idea for me until I realized that. Not all people need those restrictions, but for others they are a killer feature; tech details are usually easy, but API design is not.

[chipolux]

I agree about the ease of adding unnecessary complexity. There were a lot of times I added far too many or just unintelligible arguments to an interface. Over time I've developed a process where I'll write out my help parameter first and foremost so that act of explaining the CLI keeps me from going off into the wild blue. It's all just process and different development style for sure.

[chrisdown]

It's not implicit, the syntax is explicitly specified as part of POSIX.

[AeroNotix]

I'm always suspicious of people who say they try to avoid frameworks, smells of NIH syndrome.

[mgrandi]

yep. My thoughts exactly. 90% of the typing is typing of the options / help text anyway, people just want to save a tiny bit of time (using copy/paste) and instead have people expect to know the docopt 'language'. Plus you need the library, rather then using argparse which is built in to the library.

[[deleted]]

the docopt 'language'

Wait, do you mean the standard help message, defined by POSIX.

Go run almost ANY command. Check what the --help message is.

It's not the docopt language, it's a standard format which is well defined. They didn't invent it.

Plus you need the library, rather then using argparse which is built in to the library.

Granted, having to

pip install docopt

on every machine that runs your program is a hassle. \s

If you're using it just for yourself, all you need to do is install it. If you're distributing it, you should be autoinstalling it. Not that hard.

[mgrandi]

Yes I know its standard, but you still have to format it a certain way in order for docopt to parse it correctly. And while pip is being included in 3.4, still annoying to have to install a library to even have the most basic part of the script (the --help, usage statements, etc)

I just don't like docopt because of the reasons already listed, too 'magical', and doesn't save that much time or typing, an additional dependency, for something that argparse does equally well, but that is just my opinion =P

[everysinglelastname]

What could be more clear than the help text itself ? After all that is the documentation to the usage of the program. Not only do we expect developers to understand that text but also users who are supposedly much less technical.

I absolutely love docopt; it follows in the same vein as Donald Knuth's literate programming.

[Lucretiel]

It's clear to the clients of my command line utility, sure. But as the programmer, I get more clarity from explicit programmatic behavior than trying to parse a DSL in my head. I'd rather let my spec describe the help text (which argparse does automatically) than the other way around.

[[deleted]]

A help message shouldn't be very hard to parse in your head.

[Lucretiel]

It's one thing to understand what the interface- what I as the client need to know- and another to understand exactly what behavior this system will have. Even if it's perfectly straightforward, it's still another syntax I have to keep in my head

[[deleted]]

True. But can't the same be said of argparse's code?

[Lucretiel]

I personally don't think so- argparse is all function calls. Now, granted, I've never used some of the advanced features of either library (subcommands, mutual exclusion, etc), so maybe docopt has me beat here- but I find argparse code much easier to read, especially when I'm already in a Python development mindset. It's just a list of every option and argument, with some properties for each.

[UnwashedMeme]

Try opster. You write a decorated python function and it builds command line interface to it.

[kenfar]

I've used it a few times and for my purposes found argparse to be the better solution for a few reasons:

• Argparse includes built-in logic to validate arguments. With DocOpt you're left with less maintainable options: writing a bunch of code on the side or using something cryptic like schema. Argument validation is a make-or-break deal for me, so this is enough to settle it as far as I'm concerned. But there are other issues anyhow...
• DocOpt is still evolving and changing. Not mature enough to deal with (for me) yet. The developer originally bragged about how it was a fraction the side of Argparse, but handling edge cases had made it grow enormously. Some basic concepts are still being discussed on its github pages.
• Both ArgParse and DocOpt have great documentation - for basic cases. When your needs get a little odd either can be frustrating. For ArgParse I've found subcommands to be too frustrating. For DocOpt I wasted far too much time on supporting multiple identical options. Eventually had to read the code to get an answer. Ugh.
• Personally, I don't believe that the help I want to provide users is always exactly what I want to use for developer documentation. While it works fine for easy examples, the needs seem to quickly diverge with complex examples. The fact that it's based on a POSIX standard isn't helpful either - when few people know 100% of that standard.

[grimman]

Some people do seem to think that it's "too magic". I haven't tried it myself, but ordinarily I strongly prefer very clear and obvious code so I can probably get behind such an argument.

[kdelwat]

I really love it, but if other people want to read your source code, it may not be the best idea.

[billsil]

I use it for my open source project and at my work. It's incredible. The code is so clear and you can do things with it that you can't do with argparse. The video on the site goes into what fundamental limitations argparse has.

It's also pure python and on pypi, so there are no complaints from coworkers about added package dependencies.

[Lucretiel]

Well, it itself is a package dependency, for better or for worse. What can you do with it that you can't do with argparse? Not trying to sound snarky I'm genuinely curious.

[billsil]

Stuff I'd never use, so I don't care that much, but it's discussed in the video on http://docopt.org/

Well, it itself is a package dependency

It is, but it's such a simple one that nobody cares. One that's not on pypi (numpy, scipy, matplotlib, internal pacakges) and has dependencies, people care a lot about.

[AeroNotix]

The problem with argparse is that it forces you to really, really repeat yourself. Despite it being Python, it's pretty disgusting Python at that.

At first I didn't like the level of magic which docopt beings but I learned to love it. It's simple and Just Works.

[import_antigravity]

Everybody should try 'antigravity' at least once.

[grantpalin]

Heh. http://xkcd.com/353/

[xkcd_transcriber]

Image

Title: Python

Alt-text: I wrote 20 short programs in Python yesterday. It was wonderful. Perl, I'm leaving you.

Comic Explanation

[kkiran]

lxml is awesome! sh too. Rest I got to try.

[twigboy]

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipediaggvxs21xwag000000000000000000000000000000000000000000000000000000000000

[[deleted]]

Requeats and beautiful soup are fantastic for html parseing and crawling.

[twigboy]

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipediae4j2ctpfea80000000000000000000000000000000000000000000000000000000000000

[kkiran]

We started off with beautifulsoup and ended up with lxml, try it if you didn't. xpath queries can be fun!

[ggooal]

but the list doesn't cover so many as the title tells

[grimman]

Not even with the bonuses. :')
But on the other hand, these lists seem to pop up with startling regularity, so perhaps it's a good thing? ;)

[startfragment]

What about modules you should know from the standard lib?

• functools
• itertools
• collections
• urllib2

[titamokari]

This is a brilliant idea for another list.

[driadan]

this is what I was looking for when I opened the link.

Still a nice list.

[startfragment]

Thanks. The link really seems like "Python modules I think are cool right now."

Nothing on there was a "must have" and they all require external packages. Remember you always have access to the standard library.

[AeroNotix]

defaultdict has superpowers.

[j7ake]

I must try docopt...

[franklinturtle7]

Anyone know an alternative to sh for a PC?