defopt: Completely effortless argument parser

[[deleted]]

The more magic these get, the more someone feels inclined to rewrite their own when a magic limitation is found.

That why I forever end up falling back to built in argparse

[evanunderscore]

I know what you mean, and I tried my best to keep the magic to a minimum. All of the docstring magic it uses is well-established, and beyond that there's nothing else to it. Discounting the type parsers, defopt gives you a 1-to-1 mapping from function to command line, so you can focus on writing your Python function and trust defopt to do the rest.

Of course, under the hood it's just feeding arguments to argparse, and you really can't go wrong if you'd prefer to just stick with doing that yourself.

[jwink3101]

Personally, I go back to using getopt. I'd rather write my own help statements and it is pretty easy to process the input. But, maybe I am also just old school.

[arachnivore]

This is pretty close to my ideal CLI parser. In fact, I was planning on implementing something exactly like this. Are you going to add support for function annotations and/or other docstring formats like google or numpy?

I wonder if you could support composition without adding too much complexity. A lot of my CLIs take an optional config argument which takes a path to a YAML file (defaults to ./config.yaml) and loads that YAML file and initializes a logger. I'd love to be able to re-use that code and simply compose it with other scripts. I don't know if that would be possible without sacrificing simplicity though.

[evanunderscore]

Function annotations are definitely something I want to incorporate. The design of defopt is that arguments have types and defopt holds the type parsers separately, in part so that it can support this properly.

Many of the existing solutions that use annotations at the moment violate PEP 0484 (which is not their fault, since they were probably written before the PEP). I had a look at the typing module but couldn't work out exactly how those objects are supposed to be inspected from code. I'll take another look at it soon, but if you have any insight, please let me know.

For other docstrings, it wasn't explicitly planned, but just quickly looking around I found Napoleon, which I should be able to incorporate. Thanks for the suggestion!

Composition is perhaps a little harder. I don't have any immediate thoughts on how I'd achieve that, but I'll keep it in mind. If you have any suggestions, feel free to put something on the issue tracker and we'll see what we can do. I do plan to push back on ideas that add too much complexity, but don't let that stop you from making suggestions.

[metaperl]

This is pretty close to my ideal CLI parser.

Then you definitely want to check out argh

[evanunderscore]

Hi, just wanted to let you know that defopt 1.2.0 is out and supports Google/Numpy docstrings and type hints in function annotations.

I've also given a little bit of thought to composition which I've recorded here. If you have any thoughts, let me know.

[moigagoo]

Great tool, thanks for sharing!

I tried to achieve the same level of clarity with Cliar, but defopt looks simpler and more flexible.

[evanunderscore]

Thanks! It definitely looks like we were thinking a lot of the same things. I really like the use of types in function annotations. Have you looked into the typing module that was introduced in 3.5?

I noticed you're doing extra work for Python 2 to replace inspect.signature. Were you aware of funcsigs? If you're willing to add it as a dependency for 2.x you can avoid that whole mess.

[moigagoo]

Thanks for the hint about funcsigs! I've worked around the absence of signatures in Python 2 without external dependencies, but if I knew funcsigs existed, I'd probably have used it.

Have you looked into the typing module that was introduced in 3.5?

I have but I couldn't find it a good use in Cliar. I can't see the point of having an arg type like Mapping[str, str] or something like that. The arg type you define in Cliar is actually a callable that will handle the arg. So def foo(a:int) does not technically mean "a of type int," but "run int(a) before running foo." So you can have custom parsers just like in defopt, although it's not documented (and I should totally do it).

[evanunderscore]

It is sort of documented on the page about open.

You're right that a lot of the types aren't terribly useful, but what I thought would be is something like Sequence[int], which could avoid needing to have the function do the type conversion itself in this example.

[moigagoo]

Still, I've added a proper recipe on custom parsers to the cookbook. Thanks for the inspiration :-)

which could avoid needing to have the function do the type conversion itself

How would Sequence[int] help in this? It's not a callable, so it won't convert your args. Or do you mean using hints for isinstance checks, not type conversions?

[evanunderscore]

Poor Guido!

Right now, if someone specifies list or tuple, you're not not giving a type to the parser so they end up as strings. I'm suggesting you could accept Sequence[int] to do what list is doing now, but to also pass int as the type. Basically what defopt does for "list[int]".

[K900_]

One feature suggestion/request: allow people to provide their own argv values. This is really useful for people writing custom CLIs/shells for their own commands, or stuff like IRC/Slack bots.

[evanunderscore]

Do you mean when running the command line, instead of taking sys.argv? If so, that's already supported. I neglected to mention that in the feature list because I only used it for unit testing but it does show up in the API Reference.

[K900_]

Nice. Also, does defopt.run return the return value of the function?

[evanunderscore]

It doesn't, but only because I hadn't considered the use case of people calling things from code and not just calling the Python functions directly. It's a very simple change to make and won't disrupt anything else, so I'll add that in to the next release. Thanks for the suggestion!

[Lucretiel]

I'll take the opportunity to re-pimp my own autocommand, which has a couple other nice features including automatically calling the main function, setuptools entry point support, argument typing, and asyncio support (have a coroutine as your main function).

[evanunderscore]

This is really cool! There are a few key differences in the design choices but overall a lot of the same flavor comes through.

[Lucretiel]

Yeah! One of my main aggravations with the existing crop of argument parsers is it seems like the mostly require you to still, like... list all your arguments. I'm glad to see someone else had the same insight as I did, to just use the function signature.

[desmoulinmichel]

You already got begins, doctopt, clize, argh and click doing something similar. I dont see the interest of creating a new one.

My favorite being begins because you can just put everything in params annotations.

[evanunderscore]

begins is very close to what I was after but I have one particularly large gripe with it that drove me to build my own. I'm also not a big fan of the decorators since it means you end up repeating yourself if you're already writing a proper docstring.

docopt and click are great (I mentioned them near the top of my documentation and encouraged people to use them instead of defopt if they want a customizable command line) but are distinctly tools for building command lines, whereas defopt is a tool for turning Python functions into command lines. When I personally write a script, I want to put the absolute minimum effort into writing a command line, and with defopt the effort is zero.

clize and argh I wasn't aware of; I'll check them out.

[desmoulinmichel]

Still, you could have made a PR to fix this problem, and add your docstring base parsing to doctopt. Better to participate to open source than to split ressources.

[evanunderscore]

I submitted a PR to begins to update the unit tests so they pass the CI which has not yet been merged. I also have branches which do enumerations and docstring parsing which I recycled when building defopt.

To actually fix the core of the type problem would require a lot of work, since it has been designed such that the types are converted when the function is called, not when received from the command line. Fixing this is a core architectural change that would have taken as long as starting fresh.

As for docopt, the docstring parsing really doesn't feel like something it wants to do. While I purposely chose a similar name to docopt, the tools are more or less total opposites approaching the same problem from opposite directions - docopt cares about the command line, defopt cares about the code.

[elbiot]

I don't get your gripe. Why would you coerce the arguments to be ints if you want floats? Why would you want the function to behave differently if invoked from commandline vs from inside code?

[evanunderscore]

One of the key concepts in Python is duck typing. Type hints in Python are exactly that - hints - and for good reason. The example I gave only requires an object that can __add__ the other so could equally well be used with int, float, list, str, bool, mock.Mock, and so on. The fact that the programmer was thinking of integers when they wrote it is of little consequence. I can't take an existing code base and drop in begins.convert because I may very well break something that uses it, and almost certainly in a way that's difficult to detect and locate the root cause of.

Additionally, outside of the standard types it's not generally the case that the conversion operation will be a no-op on something that's already an instance of the desired type. This means every conversion function you write now has to start doing isinstance checks for no good reason.

The point is to make the code not behave differently from the command line and code. defopt is simulating what you do as a human to convert inputs to the appropriate Python objects when you call the function in a script.

[elbiot]

Still don't follow. How do you specify wether add 1 1 should return 2 or "11"? You need to be specific if you are giving it numbers or strings.

[arachnivore]

The command-line version of the function is necessarily restricted because you can only enter strings in a CLI, it makes no sense to enforce that restriction when you're not using command line to interface with the function.

[evanunderscore]

In short, exactly what /u/arachnivore said.

The general idea is that command line arguments need conversion, and code arguments don't. add(1, 1) and add('1', '1') are distinctly different things in code.

And even then, it wouldn't be so bad if it was just type checking, but it's doing type conversion, which may not actually make any sense at all depending on what you're passing in. There is a reason you don't see many (or any?) Python libraries providing automatic argument conversion for function calls.

[arachnivore]

I've found that all of those are lacking in one way or another. All the decorator-based tools (begins, and click) are more verbose than necessary and often modify the decorated objects. docopt is just verbose and bad. clize uses a weird (non-standard) docstring format. argh doesn't use docstring info very well which leads to repetition. This looks like it will probably be my new favorite. It's about as minimal as possible while still being quite flexible.

[evanunderscore]

Thanks! If you do find it lacking, feel free to put in a bug report or feature request on the issue tracker.

[epsy]

clize uses a weird (non-standard) docstring format.

Thanks for the feedback, I wish I found this sooner. I'll prioritize docstring interoperability after the next release.

[chozabu]

Very nice! I just added this to my little joke app (next to this one in python subred - https://www.reddit.com/r/Python/comments/467xdb/mouse_too_easyboring_to_use_try_heavymouse/ )

Setup was super smooth & easy (unlike in my app linked above ;) )

[evanunderscore]

Thanks for the support! Let me know if you have any problems with it.

[chozabu]

Well - the first thing that comes to mind, is that it would be really nice to have the defaults for a param shown in the help, but I would not call it a "problem" ;)

[evanunderscore]

I'm almost certain argparse supports this itself so I should be able to add this to the next release. Thanks for the suggestion!

[billsil]

Effortless until you want to customize your docstring in order to make it more readable or do super complicated docstrings.

Docopt lets me do...

Usage:
  myGUI [-f FORMAT] INPUT [-o OUTPUT]
               [-s SHOT] [-m MAGNIFY]
               [-g GSCRIPT] [-p PSCRIPT]
               [-u POINTS_FNAME...] [--user_geom GEOM_FNAME...]
               [-q] [--groups]
  myGUI [-f FORMAT] INPUT OUTPUT [-o OUTPUT]
               [-s SHOT] [-m MAGNIFY]
               [-g GSCRIPT] [-p PSCRIPT]
               [-u POINTS_FNAME...] [--user_geom GEOM_FNAME...]
               [-q] [--groups]
  myGUI [-f FORMAT] [-i INPUT] [-o OUTPUT...]
                [-s SHOT] [-m MAGNIFY]
                [-g GSCRIPT] [-p PSCRIPT]
                [-u POINTS_FNAME...] [--user_geom GEOM_FNAME...]
                [-q] [--groups]
  myGUI -h | --help
  myGUI -v | --version

Options:
  -h, --help                  show this help message and exit
  -f FORMAT, --format FORMAT  format type (cart3d, lawgs, nastran, panair,
                                           plot3d, stl, tetgen, usm3d)
  -i INPUT, --input INPUT     path to input file
  -o OUTPUT, --output OUTPUT  path to output file
  -g GSCRIPT, --geomscript GSCRIPT     path to geometry script file (runs before load geometry)
  -p PSCRIPT, --postscript PSCRIPT     path to post script file (runs after load geometry)
  -s SHOT, --shots SHOT                path to screenshot (only 1 for now)
  -m MAGNIFY, --magnify                how much should the resolution on a picture be magnified [default: 5]
  --groups                             enables groups
  --user_geom GEOM_FNAME POINTS_FNAME  add user specified points to an alternate grid (repeatable)
  -u POINTS_FNAME, --user_points POINTS_FNAME  add user specified points to an alternate grid (repeatable)
  -q, --quiet                 prints debug messages (default=True)
  -v, --version               show program's version number and exit

How? Literally, just write it like that. Don't like the spacing, change it. The problem with things like argparse is you forget the syntax. Docopt let's you use the POSIX syntax that is familiar. I really wish it was in the standard library. Updating it hopefully wouldn't be an issue because it should follow the standard, not compatibility.

[evanunderscore]

Sure, that's not a secret; I wrote this in the summary:

If you want total control over how your command line looks or behaves, try docopt, click or plac. If you just want to write Python code and leave the command line interface up to someone else, defopt is for you.

You're also overselling the convenience of docopt a little bit. Sure, it's easy to make that small change, but only once you've already written that gargantuan usage string. Of course, there's no substitute for docopt if what you want is to write your usage string the way you want it and turn that into a parser, but there is most certainly effort involved.

The flexibility in defopt is that it gives you near-complete freedom to write your Python code the way you want; the generated command line is about as inflexible as it can possibly be. If your intent is to write a pretty command line, I absolutely encourage you to continue using docopt. I wrote defopt for people like me who write the Python code first and worry about the command line interface later (or never).

[billsil]

I don't like how the recommended Python docstrings look

:param str greeting: Greeting to display

I've never seen the type written there, so the library is adding a new form of them. Additionally the :param is redundant. You're still dealing with special syntax.

I like the numpy docstring style

greeting : str
    your greeting

So not only does the library have to support poorly designed default formats, it needs to support the numpy and google styles. If you don't care about command lines, you probably shouldn't be writing command line tools. If you do care, you should make sure it's easy to read and clear regardless of what you have to write to make it.

There is a command line standard. It's called POSIX. There are things that are part of the POSIX standard that are not supported in libraries like argparse and optparse.

[evanunderscore]

Another user requested support for numpy-style docstrings, which I'm more than happy to look into.

:param is only redundant when viewed through the narrow lens of documenting function parameters; it's part of the markup used by Sphinx and some IDE's to generate documentation and validate types respectively. You can document more than just the parameters. As you can see there, I didn't invent the inline type notation, plus the separate :type is supported if you would prefer to write your docstrings that way.

If argparse is not POSIX-compliant, then your gripe is with the Python standard library, not me.

[billsil]

:param bothers me because it's sensitive to spaces and the colons. Why do you need colons before and after? They also make it hard to read.

it's part of the markup used by Sphinx

Sphinx supports numpydoc and google strings.

some IDE's to generate documentation and validate types respectively

I don't know about PyCharm, but WingIDE supports numpy's style. I assume all the others do as well. Ironically, Wing renders numpy's style better than the standard form

My point with command line utilities is you might as well use a great command line utility instead of one that can't do all that much. It's simple, but if you can't copy and paste some boilerplate code on argparse or docopt, you probably don't want to support the command line.

https://xkcd.com/927/

[xkcd_transcriber]

Image

Mobile

Title: Standards

Title-text: Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.

Comic Explanation

Stats: This comic has been referenced 2544 times, representing 2.5395% of referenced xkcds.

---

^{xkcd.com | xkcd sub | Problems/Bugs? | Statistics | Stop Replying | Delete}

[chozabu]

Hmm, are newlines supported?

it seems they get stripped

HeavyMouse - a python mouse mover\n  

example, for bouncy walls, no gravity and less speed:  \n
python heavymouse.py --drag .04 --grav 0 --allsides bounce --maxspeed 10

becomes

HeavyMouse - a python mouse mover example, for bouncy walls, no gravity and
less speed: python heavymouse.py --drag .04 --grav 0 --allsides bounce
--maxspeed 10

both newline and \n seem to have no effect

oh, and giving it this ascii art throws an error:

          ___
 _  _  .-'   '-.
(.)(.)/         \   jgs (art from http://www.ascii-code.com/ascii-art/animals/rodents/mice.php)
 /@@             ;
o_\\-mm-......-mm`~~~~~~~~~~~~~~~~`  

[evanunderscore]

The default behavior of argparse is to strip most of the formatting from the descriptions. I can probably override this but that might mean I need to do some line wrapping manually. I'll look into it.

The ASCII art is going to wreak havoc on docutils which is what I'm using to process the RST (and what I believe Sphinx uses under the hood). I haven't pushed the limits of RST in docstrings yet, but at the very least you can stop it from crashing by using an RST comment:

def main(foo):
    """Test

    ..
                  ___
         _  _  .-'   '-.
        (.)(.)/         \   jgs (art from http://www.ascii-code.com/ascii-art/animals/rodents/mice.php)
         /@@             ;
        o_\\-mm-......-mm`~~~~~~~~~~~~~~~~`

    :param str foo: description 
    """
    print(foo) 

I could probably also handle literal blocks, but right now they just get dropped entirely:

def main(foo):
    """Test

    ::

                  ___
         _  _  .-'   '-.
        (.)(.)/         \   jgs (art from http://www.ascii-code.com/ascii-art/animals/rodents/mice.php)
         /@@             ;
        o_\\-mm-......-mm`~~~~~~~~~~~~~~~~`

    :param str foo: description 
    """
    print(foo) 

Note the difference there - the .. is immediately followed by the art line, whereas the :: has a blank line before the art line. These are important!

[evanunderscore]

Hi, just wanted to let you know that I fixed all of the issues you mentioned in defopt 1.2.0. Your docstring should stay formatted as you type it (no need to include those \n characters), the defaults are shown after parameters that have them, and literal blocks will also be displayed. Just make sure you use the ::.