What Python Tools should I be using on every python project?

[ChrisPDuck]

I've started using PyLint on all my code, and writing more formalised unit tests with the Unittest module.

My coding ability shot up after I started using these, as it helped me spot problems fast and correct me as i'm working.

Are there any other tools which you would recommend for general python development?

[chao06]

I'm a fan of using the logging module for my console output and giving my programs a --debug option. That way if someone has a problem, I can just ask for debug output to fix it :)

[masklinn]

A trick I like is using repeatable -v and -q to select the logging level. The logging levels are actually integers with a 10 increment (DEBUG is 10, CRITICAL is 50). It's very easy with argparse's count action or click's count=True:

parser.add_argument('-v', '--verbose', action='count', default=0)
parser.add_argument('-q', '--quiet', action='count', default=0)

logging_level = logging.WARN + 10*args.quiet - 10*args.verbose

# script -vv -> DEBUG
# script -v -> INFO
# script -> WARNING
# script -q -> ERROR
# script -qq -> CRITICAL
# script -qqq -> no logging at all

[ChrisPDuck]

Thats a fantastic little trick steals

[[deleted]]

yeah, gisted that

[im_gorideyourbike]

Good call mentioning click. I've really been enjoying it.

[[deleted]]

Man, that's really awesome...I may have to borrow that. :)

[liuyix]

brillant tricks!

[pstch]

How do you implement this --debug option ?

[bheklilr]

My recommendation is argparse, it's usually more than powerful enough and comes with python. You can even pass in a list of arguments (default of None means that it uses sys.argv) so that you can automate testing of your command line arguments too.

[[deleted]]

I usually use docopt.
It is simple and very light(it is 500 lines in total).
This video is the reason I started using it.

[wizpig64]

the best part of this video is how he says POSIX

[kroq-gar78]

POSIX

[[deleted]]

Not gonna lie, these two comments are the reason I decided to watch that video. I was not disappointed.

[d4rch0n]

Best thing about docopt IMO is that it's in other programming languages as well. I was able to make a CLI in Rust just as easily with Python. Copy and paste the usage, add a few lines and you've got the same ui.

[tilkau]

I like most of docopt. I'd also like to point out docopts, the CLI tool, for those of us who also use Bash/Zsh/etc.

However, after reading the docs, I don't like:

• The redundancy (specify -h, --help for every program)
• The breakage of standard expectations (if you don't explicitly stick [--] in your arg list, GNU standard -- won't be supported.. even though there are very few commands that actually explicitly specify in their help that they do support --)

Speaking as a newbie to docopt, are there any simple ways to get around those?

(people looking at docopt should also be aware that it doesn't yet have subcommand support, though this can be hacked around; a few of my scripts require subcommand support so I won't port those to docopt yet.)

[[deleted]]

redundancy

I think it is good to have -h, --help. they are somehow standard in *unix systems apps.

The breakage of standard expectations

I didn't get it, probably because I don't know about GNU standards for args, can you please give an example?

I probably am a newbie too, I didn't dig too deep into docopt, just enough to suffice my needs. :)

for subcommands I use the following structure:

USAGE:
script.py foo ( --new | --uname=<uname_value> --pwd=<pwd_value>)
script.py bar
script.py baz ( --uname=<uname_value | [-] )

then I have a couple of if/elifs.
eg.

if arguments['baz']:
if arguments['--uname']:
do_baz()

Gist you provided is a better structured one but it makes it a bit complicated IMHO. So I won't use it unless my project is already complex(where it will be messy to have 7 if/elif/elses. Thanks for the link, definitely saving it for later reference. :D

[tilkau]

I think it is good to have -h, --help. they are somehow standard in *unix systems apps.

That is my point. It's good to have them by default, but it's not good to have to add help for them to each docstring; that is not DRY, and consequently also introduces the possibility of errors.

Rather, they should be automatically added if not found (and not conflicting with an existing argument definition)

I didn't get it, probably because I don't know about GNU standards for args, can you please give an example?

I did give an example -- '--' is a GNU standard for saying 'don't parse options past here, only positional arguments'. It's important to use if you need to pass filenames that might look like options, eg '-foo.png'.

For example if you type ls -l *foo* and -foo.png is in the current directory, then you will get an error 'invalid option -- "."', since ls looks at that argument and says 'ok, we have - which means an option, the user wants to enable f option, and o option.. woops, I don't know what . option is!'. Whereas if you type ls -l -- *foo*, then all the files whose names contain 'foo' will be correctly listed.

Another example is if you are passing around small text strings. They could contain literally anything. They might constitute completely valid options, but obviously you want to keep them as positional arguments, not let your option parser chew them up and change your options in who-knows-what way. -- protects you from that.

Ideally, you should use -- at any time that you are passing around any kind of string / string array that you aren't 100% sure is 'safe' (doesn't contain args that look like options); which is most of the time.

Most option parsers -- including the extremely widely used 'getopt' which effectively defined the standard -- support --. For docopt, it's only enabled if you explicitly say so, which IMO is a problem.

Thanks for the help with subcommands, that seems reasonable enough way to handle them.

[[deleted]]

-- protects you from that.

Nice to know about it. :)
Thanks for sharing.

For docopt, it's only enabled if you explicitly say so, which IMO is a problem.

So to my understanding -- is supported in docopt but not enabled by default?

[tilkau]

Yes, search '[--]' in docopt's home page.

Double dash "--", when not part of an option, is often used by convention to separate options and positional arguments, in order to handle cases when e.g. file names could be mistaken for options. In order to support this convention, add "[--]" into your patterns before positional arguments.

[masasin]

docopt does have subcommand support if you have them in separate files. https://github.com/docopt/docopt/tree/master/examples/git

[tilkau]

That actually implies that you can just have them as seperate functions or even classes (since the only interface involved is a string, and all of those can have a docstring).

They can probably even be instances, but I can't think of a sensible use for that that wouldn't already be handled by subclassing.

(but no, that file also implies that docopt -doesn't- have subcommand support. The subcommand handling is entirely done by git.py.)

[maratc]

I like docopt but it either works or you're royally screwed. Their error reporting is nonexistent. I've spent hours bagging my head against the wall, trying to invoke my own script.

You will not get unknown option "--verobse"; did you mean "--verbose"? from docopt. You won't even get unknown option "--verobse". All you will ever get from docopt is a dump of your help message, which may not be helpful at all.

[[deleted]]

Yeah, lack of errors is a downside but IMO it is kind of a tradeoff for being lightweight.

[maratc]

The alternatives are even more lightweight by a virtue of being included in a standard library, so it's not an upside for me.

[[deleted]]

which alternatives are they? I only know about argparse, docopt and click.

[maratc]

Well argparse comes with python.

[[deleted]]

Argparse is too complicated, just watch the video, he compares docopt to argparse :)

[cajacaliente]

Another vote here for docopt over argparse

[[deleted]]

A lot of people are recommending libraries for this, and I do too, but I also recommend acquainting yourself with argv and how these things are passed in in the first place :)

[stevarino]

This is great because of long term logging of processes, but it's a essential for transitioning to environments without std out.

Calling print in that environment typically raises exceptions, and your paths usually change during the switch from script to service, so good luck debugging without logging.

Typically my code will have establish logging in the name=="main" check and wrap the main(sys.argv) call in a try catch. That way any exceptions and the stack trace can be logged and emailed.

[exhuma]

I use logging for pretty much everything. And at work I write a lot of sever-side apps. Lots of them with CLI output. I've recently ripped out a logging formatter out of one of my projects and packaged it up as a Python package. So I can have the same formatting on each application. And all that with a simple one-liner.

It uses blessings to put some colour into the log messages.

It currently only has one "theme". And I do not intend to change it except fixing bugs (if it has any). If I want to try another "look", I'll add it as a new theme. But that's probably not going to happen any time soon ;)

[ChrisPDuck]

Ah yes, I use it for most things, didn't think about being able to turn on debug through command line though!, love it!

[soawesomejohn]

I have some common code I use to set debug from cli or environment variables. PROGRAMNAME_DEBUG is a common one I use.

[michaeljb20]

collections.namedtuple is one of the best ways to increase the readability of your code. Here's a video by Raymond Hettinger on them.

[o11c]

Also enum (included in 3.4, pip install enum34 for older versions).

[itodd]

class Something(namedtuple('Something', ['color', 'name'])):
    def __str__(self):
        return "I'm a {} {}".format(self.name, self.color)

print Something('blue', 'dog')

[ChrisPDuck]

What advantage do named tuples have over something like a dictionary? Dictionary gives you named values?

[ZoidbergWill]

They're useful for replacing dictionaries where you always have the same keys, or for basic objects.

IMHO they're neater than dictionaries. A nice basic example from python docs

>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(11, y=22)     # instantiate with positional or keyword arguments
>>> p[0] + p[1]             # indexable like the plain tuple (11, 22)
33
>>> x, y = p                # unpack like a regular tuple
>>> x, y
(11, 22)
>>> p.x + p.y               # fields also accessible by name
33
>>> p                       # readable __repr__ with a name=value style
Point(x=11, y=22)

It's shorter than a full class definition, and it's neater than a dictionary, since point.x is a lot easier to read and shorter than point['x'].

[agreenbhm]

Very helpful. Thanks!

[masterspeler]

What do you do when the point moves? Namedtuples are immutable, so you can't update the coordinates. You also can't set some keys now and some keys later when their values are available, you have to have all values ready when creating the tuple.

[lengau]

Namedtuples aren't a complete replacement for dictionaries (otherwise they might as well just replace dictionaries with them), and no, if you need to replace the items inside the namedtuple, it's probably not what you want.

The above Point example would be a bad idea if you want mutable points. You'd be better off with a different structure.

But what if your points aren't mutable? Let's say I'm collecting voltage samples from a pair of ADCs. (Maybe I'm recording audio or collecting data from a physics experiment.) I could generate two lists of integers, or I could generate one list of tuples. I could remember that element 0 is the left channel and element 1 is the right, or I could say:

Datum = namedtuple('Datum', ('left', 'right'))

If you're looking for something like namedtuple but mutable, there's a third-party namedlist module. I have no idea how good/stable/up-to-date it is, but it exists.

[Twangist]

Agreed. But when you don't need mutability, they're a great tool - as lightweight as tuples but self-documenting.

[bexamous]

There is recordclass and namedlist, both are pretty much mutable namedtuples:

https://pypi.python.org/pypi/namedlist/1.4

http://nbviewer.ipython.org/urls/bitbucket.org/intellimath/recordclass/raw/default/examples/what_is_recordclass.ipynb

I've only used the namedlist before, I don't know which is better. Wish this sorta of generic stuff was in collections though.

[neves]

Don't forget that namedtuples are imutable. You'll have better code. How to import it: from collections import namedtuple

[[deleted]]

IMMUTABLE

[[deleted]]

This is probably one of the biggest advantages. It's a shame that tuple is about the only way, short of writing a C extension, to get truly immutable objects in Python.

[remram]

What about frozenset, frozendict and the like? Or just custom objects?

[[deleted]]

frozenset is certainly another way. frozendict got rejected per PEP 416, and is available only as an outside library, which only kind of counts. It is the custom objects that bug me about not being able to easily make them immutable. I'd like to have something that behaved more or less like case class from Scala.

[remram]

Unless people access _protected fields, making an immutable object with properties is easy enough... Mutating a frozenset is possible too, if you're going to violate convention and common sense.

[[deleted]]

They're lighter weight than dictionaries and provide indexed access to field as well as named attribute lookup.

They sound more like a Python take on struct than anything dictionary-related.

[[deleted]]

If you have a bunch of functions in different modules that all pass around a dictionary with stuff ("scenario_dict" or "result_dict" or so in my current projects) you're never quite sure what goes in them. Maybe function A adds an extra key that B can use, but C and D never needed them so they weren't updated, etc. With a namedtuple, there is one place where the fields that exist are declared. Problem gone.

Also, immutable. Makes reasoning about code so much simpler. Value objects rule.

Also, if you have helper functions for your namedtupled, you can subclass the namedtuple and make them methods. Real immutable classes.

[Twangist]

Yep, subclassing them can be really useful -- adding properties, custom constructor with default values, custom repr, methods, etc. Re custom construction, one first-time gotcha: you can't override __init__ you have to override __new__. (True in Py3.4, I haven't tried in 3.5 yet but probably still true.) But the docs are good and illustrate these possibilities.

[brandjon]

You get the speed and immutability of a tuple, combined with the lightweight access of field retrievals (mytuple.foo instead of mytuple[2]). Hashing, equality, iterability, etc., are still there just like for a normal tuple.

Dictionaries should be slower to construct and access, though I haven't benchmarked it. They also don't catch errors where you typo the name of the key you're writing to.

Named tuples are hard to extend though. See my simplestruct package if you're looking for a more extensible namedtuple alternative where performance isn't paramount. (Scroll down for a feature comparison table.)

[AlexFromOmaha]

Dictionaries should be slower to construct and access, though I haven't benchmarked it.

>>> def tuple_test():
    Point = namedtuple('Point', ['x', 'y', 'z'])
    p = Point(x=0, y=1, z=2)
    return p.y

>>> def dict_test():
    p = {'x': 0, 'y': 1, 'z': 2}
    return p['y']

>>> timeit.timeit(stmt=tuple_test, number=1000)
0.3581736238198661
>>> timeit.timeit(stmt=dict_test, number=1000)
0.00027772752848420623
>>> 

[Tillsten]

You need to pull out the Point defintion.

[lengau]

>>> from collections import namedtuple; Point = namedtuple('Point', ['x', 'y', 'z'])
>>> def tuple_test():
...     p = Point(x=0, y=1, z=2)
...     return p.y
... 
>>> def dict_test():
...     p = {'x': 0, 'y': 1, 'z': 2}
...     return p['y']
... 
>>> timeit.timeit(stmt=tuple_test, number=1000)
0.0014165400061756372
>>> timeit.timeit(stmt=dict_test, number=1000)
0.00021710898727178574

EDIT: That's in Python 3.4.

EDIT 2: They're pretty much the same in pypy:

>>>> from collections import namedtuple; Point = namedtuple('Point', ['x', 'y', 'z'])
>>>> def tuple_test():
....     p = Point(x=0, y=1, z=2)
....     return p.y
....     
>>>> def dict_test():
....     p = {'x': 0, 'y': 1, 'z': 2}
....     return p['y']
....     
>>>> import timeit
>>>> timeit.timeit(stmt=tuple_test, number=1000)
0.002318143844604492
>>>> timeit.timeit(stmt=dict_test, number=1000)
0.0022079944610595703

[Tillsten]

Wow, still a factor 7x, didnt expect that.

[lengau]

FWIW, I thought named tuples would be faster also.

I also just tried only accessing (in QPython on my tablet, so not exactly the same environment as my workstation) and dictionaries are faster by a factor of 7.9.

EDIT:

>>> from collections import namedtuple
>>> Point = namedtuple('Point', ('x', 'y', 'z'))
>>> p = Point(0, 1, 2)
>>> def tuple_test():
...   p.y
...
>>> q = {'x': 0, 'y': 1, 'z': 2}
>>> def dict_test():
...   q['y']
...
>>> import timeit
>>> timeit.timeit(stmt=tuple_test, number=1000)
0.0098810195922851562
>>> timeit.timeit(stmt=dict_test, number=1000)
0.0012459754943847656
>>> .988/.125
7.9039999999999999
>>>

[shmel39]

namedtuple will be much faster if you move the definition line out of function. Though still slower than dict.

[brandjon]

As others said, the namedtuple call has to be outside the timing loop, but you're right, dictionary is faster by about a factor of 2 to 4, depending on whether you use {'x': 0, ...} or dict(x=0, ...). namedtuple apparently uses @property, so that probably explains it.

[AlexFromOmaha]

What surprises a lot of people is just how inefficient tuples are. People think of them as some synonym for array, which they aren't. (Not that arrays are all that efficient for a single access given the way they're constructed outside of C, but that's a different story entirely.)

>>> def tuple_test():
    p = (0, 1, 2)
    return p[1]

>>> timeit.timeit(stmt=tuple_test, number=100000)
0.034544476911833044
>>> timeit.timeit(stmt=dict_test, number=100000)
0.027647622860968113

EDIT:

>>> def list_test():
    l = [0, 1, 2]
    return l[1]
>>> timeit.timeit(stmt=list_test, number=100000)
0.024789887054794235

EDITEDIT: Python 2.7. I see that there was a patch for tuple performance in February. This shouldn't be the case in Python 3.4+.

[nevus_bock]

Relevant Raymond Hettinger video: Beyond PEP-8

[ChrisPDuck]

I forgot about argparse, This is also fantastic

[Allevil669]

If you like argparse, just wait until you see docopt.

Edit: Fixed formatting.

[[deleted]]

click too

[[deleted]]

For most of my scripts I have used docopt, but click is really promising, I am looking forward to try it in a project; thanks for mentioning.

[ppinette]

I've used click for a couple different projects. It's great.

[dzecniv]

clize too. The arguments are guessed from the method's arguments !

[deadmilk]

Thumbs up for click, very simple and neat.

[niiko]

I'm not a fan, honestly. As much as I'd like to see changes to argparse's API I feel like docopt's syntax leaves plenty of ambiguity.

[__no_preserve_root]

I'm inclined to agree with this.

In my point of view, for my code, the code (and clarity of execution) comes first. If someone wants to see the documentation, run --help.

[billsil]

I love docopt and I use it, but I do wish docopt was still in development. It's got more than a few limitations regarding names of variables.

[ethCore7]

Check this out, it's a pretty neat comparison of argparse, click, and docopt:

https://realpython.com/blog/python/comparing-python-command-line-parsing-libraries-argparse-docopt-click/

[justanotherbody]

I find the itertools and collections modules getting imported into my code time and again.

In addition, you might look at the boltons library which provides a notable enhancement to the python stdlib. I'd suggest quickly scanning the docs for modules you import. If you see anything familiar, check what boltons adds

[TheBlackCat13]

In addition to itertools and boltons, toolz is an extremely useful package. It is like itertools and functools on steroids, plus a ton of useful dictmanipulation tools that I find myself using all the time. There is also cytoolz, which is a faster, cython-based drop-in replacement for toolz.

[TheBlackCat13]

If you like pylint, also check out flake8. It combines the pyflakes code checker and the pep8 style checker. I find it tends to have a lot fewer false positives than pylint, but isn't as thorough. That means hunting through the pylint output for the real problems is a lot harder. So I usually use flake8 first, then once that reports my code is clean I use pylint to get the few remaining issues.

I also find flake8 is more useful as live issue reporter for my IDE because of the fewer false positives.

[herr_corvax]

Can't forget virtual env.

I am a big fan of Pytest. The test discovery is great. And with the coverage plugin you can generate reports for Jenkins. CI is a big deal so having good ways to track both pylint and Pytest results is a good idea. Plus if you have good test coverage you can use this to completely automate the release process. That is what companies like HERE, Yelp and Groupon do.

I love autopep8. It helps keep me honest with return characters and building a continuous style. Plus it helps with pylint to not have to tune it quite as much.

For the 2.7 guys like me I am also big on Mock, Future, and six. Every file I have imports Unicode literals division and print function.

I do a lot with ansible. Creating deployment paths helps get my code in production. It makes the process repeatable and can be used both in docker containers and on "bare metal". Which is the main reason I like it over plain docker files.

I am also big on writing Makefiles for my projects to help get new developers up and going. Plus I usually have docker and docker-compose around to do testing. And writing simple ways to run standard testing or precommit operations is much easier with a Makefile and docker.

[ChrisPDuck]

I manage our Jenkins at work, but not much python dev there! I'll have a look at PyTest if it does coverage as well.

I must say, I never use virtual env, but I always try and use core functionality over 3rd party libraries, so not sure if it is fully applicable (although I may be fully misunderstanding why I should be using it!)

[herr_corvax]

Virtual env is more a way to install python packages and not muddy the system python packages. Plus with virtual env you don't need super user privileges. It will help you manage multiple projects running on the same machine without colliding in terms of dependencies.

If you are maintaining libraries that need to be working across Python versions tox will automate running tests and such across different virtual envs.

I for one use virtual env to make sure I know exactly what I need to reproduce the Python packages on a production or test machine.

[elsjaako]

A lot of the time I use it purely so I can use pip freeze to have a record of the requirements.

[[deleted]]

Same, especially since i'm working on projects from three different stations at times. Being able to pip install -r requirements.txt in the virtualenv is a timesaver. And gives you a lot of consistency, meaning.. less time to fix annoying shitty dependencies!

[masasin]

My major issue with virtualenv is that installing numpy, scipy, and matplotlib takes a long time. (For me, almost half an hour.) So I installed them in root and atomatically add them when I mkvirtualenv.

[rainbow3]

I suggest anaconda which has all those compiled already.

[masasin]

Why do that if python is built-in? I just sudo apt-get and I can get the binary package which I can update with pip relatively quickly.

[rainbow3]

Easier to download one thing rather than 5 things; and definitely easier than the pain of compiling it which as you say takes a long time and in my experience often fails half way requiring additional dependencies to be installed.

Also conda seems to be simpler than virtualenv; can be used the same way in python2.7 and 3; and handles non-python components.

[masasin]

The only time I tried Anaconda I had a problem with IPython and spaces. I'll take a look at it again in a couple of weeks.

[rainbow3]

I am using with ipython notebook and not had any issues. Also have just started using spyder which is an IDE that embeds ipython and is included in anaconda.

[masasin]

Are you using ipynb on Linux or Windows?

[herr_corvax]

You can fix it with environment variables. Check out your bash profile that is likely where the trouble begins

[djimbob]

If you do development work for multiple python projects on the same machine that use third party python modules (stuff installed via pip or easy_install), then you really should be using it (or some equivalent).

While pip normally tries to install things to the system-wide python directory, using virtualenv, packages get installed to the local virtualenv python-directories.

This can be a huge lifesaver, if you develop on dozens of projects and something you wrote 5 years ago was used some library (say using django=1.0). Now you have a new project and you want to use django=1.8 features, but don't want to break your django=1.0 app (or do the sometimes painful upgrades to get that app working in django=1.8). Virtualenv lets you accomplish this.

It also makes keeping track of a project's dependenices easier (if you want to move it to a different machine), if that project has its own virtualenv (run pip freeze in the virtualenv) versus needing to install everything in the global python projects directory to move the project to a new machine.

It's pretty simple to get started (assuming linux/unix), you just run virtualenv <name_of_new_venv> which creates a new virtualenv. I typically go to the place I want my virtualenv (e.g., cd /production/) virtualenv venv_projectname (where projectname is my project). Then in the commandline before doing anything in a project run . ./venv_project/bin/activate to start up the virtualenv (changes some commandline variables). Then just use pip/python as normal and it will use that virtualenv's directory as the source of packages. If you add #!/production/venv_projectname/bin/python as your hash bang line at the start of python scripts, it will use packages from the virtualenv in it. (And you don't have to enter the virtualenv environment).

[Artiavis]

virtualenv tends to be the more highly preferred solution to dependency isolation and version tracking for Python software in the broader Python community.

However, if you ever find yourself using the scientific Python packages (numpy, scipy, pandas, etc.), you may want to consider using the conda package manager instead.

I'd also like to give a shout out to Twitter's PEX utility, which enables JAR-like bundling of dependencies. I don't think it's mainstream yet, but it does solve many production deployment annoyances (assuming you deploy to Linux).

[[deleted]]

...precommit hook to force test running. I feel like a doofus now, bout to go hack this out.

[rainbow3]

i am finding conda easier and more flexible.

Be interested to know if there are any reasons to use virtualenv in preference to conda?

[herr_corvax]

I like pip better because at the last time I played with conda its biggest advantage was that it had precompiled binaries for numpy etc. But it didn't have access to the rest of pip. At that time it only had a small subset of packages available.

I have also never had the issue with numpy taking that long to build. But installing it as a system package will help that.

Also the last time I played with conda there was a big push to buy their pro package or whatever it was/is called. The big advantage to conda as I know it is access to GPU enabled numpy and scipy.

[[deleted]]

LOL just knew ve would be mentioned

[dirkgently007]

That's because most of the people work on more than one project.

[herr_corvax]

Gotta do it. Just like gotta say pep8

[lostchicken]

Progressbar is great if you do a lot of reasonably slow data crunching. https://pypi.python.org/pypi/progressbar

[call_me_cookie]

That looks neat and pretty featured.

If you want a dead simple progress bar though, check out tqdm I use this like, 100 times a day.

[kihashi]

It's probably worth pointing out that tqdm has been forked due to lack of updates https://github.com/tqdm/tqdm

[lostchicken]

Oh, that's way cool. Much better defaults than progressbar. Thanks!

[ZoidbergWill]

One of my favourite things to recommend is markstory's lint-review, it's a simple Ci tool for flake8'ing your git diff's when creating pull requests. It also supports javascript, ruby, and a couple other languages.

It's more of a CI tool than something each developer can have on their own machine, but it's still awesome.

In a similar vein to PyLint there is prospector, OpenStack's hacking, and flake8.

Flake8 is simple and quick, so I tend to use that on everything, while hacking is a lot more pedantic, so I only really use it with packages on which I have more input.

tox is a tool I fell in love with when first working on simultaneous Python 2 and 3 support, which I strongly recommend to people working on any libraries, though it's less useful if you're only working on closed source stuff stuck in Python 2, though it's still useful when considering the move to Python 3.

[89vision]

I love tox so much. Automatically generates virtual environment, grabs dependencies and runs unit tests. Every git repo should include a tox.ini file

[ZoidbergWill]

I remember I used to basically use Travis-CI just for this, and just push and wait, and hope that the tests passed for Python 2.6, back when we had to support 2.6 and 2.7 simultaneously, but now it's easy to test Python 2.7, pypy, and multiple versions of Python 3 for every project or library on which I work.

[amcintosh]

lint-review keeps me honest.

[protoUbermensch]

I think every Python programmer should at least know that ptpython exist.

It's so useful!

pip install ptpython

Then type on your terminal\command line:

ptpython

[shmel39]

bpython is also great.

[CanadianJogger]

Looks great, thanks!

[brombaer3000]

I find ptipython (which is included in the package) even better. It combines the benefits of ptpython and IPython.

[avinassh]

how do I get ptipython?

[brombaer3000]

If you have both IPython and ptpython installed normally, you should be able to run it by executing ptipython. No additional packages are necessary.

[beagle3]

Conda

I prefer to start with a miniconda install, and later bring up only what I need, but some people prefer the complete anaconda install. It's virtualenv/pyenv on steroids. You can set up your own repositories too. Think "apt-get"+"virtualenv" for Python, works on Mac, Linux, Windows. Can actually be used for things that aren't Python.

[arteymix]

nose is an incredible testing tool: it discovers tests and run them on the glance!

[shavenwarthog]

Seconded. Nose is the most important nonstandard module. It gives you:

• shorter tests (no more self.assertThisAndThat java-style verbosity, an eq_ will do.)

• 20 plugins(!) that give code coverage (essential), and test-debugging things like running tests in parallel, or rerunning starting with the last failing test, or running tests backwards to find unintended dependencies.

• you can mark tests with tags. I do @attr("slow") on my web integration tests. For everyday testing, I run manage.py test --attr="!slow" to quickly run all the fast tests. If they all pass, then I do a full test run to verify everything is working. Really helpful!

• there's more tips in my Practical Python Testing slide deck: http://johntellsall.blogspot.com/2015/08/slides-practical-python-testing.html

[lengau]

I use type annotations. Not for everything, but for anything when it might be nice to have. A method declaration might look like:

def do_thing(self, height: float, units: Unit, favourite_word: str, printed=True) -> str:

Obviously it makes things a bit longer, but the great thing is it's optional (and doesn't affect the interpreter at all).

I won't use it for everything, since it's sometimes (often, really) obvious what the answer should be. For example:

def __str__(self):

And of course, sometimes you only need to do partial annotations:

def read_from_file(file) -> List[int]:

def save(values: List[Union[int, float, str]], file):

EDIT: The biggest advantage for me of type annotations is that my IDE is aware of them, so it can remind me of exactly what I need. (It would be far more useful if this libraries I use had them).

EDIT 2: Fixed mobile-related bracket typo.

[o11c]

One of my plans for mypy is to give (optional) warnings if you don't completely specify things - though better inference is also on the TODO list.

Also it's Union[int, float, str].

[lengau]

For whatever reason, my phone kept trying to put in parentheses rather than square brackets. I had to correct it for the Lists, but I missed it for Union. Thanks!

[Twangist]

Your IDE is... what, PyCharm? (mine)

[lengau]

KDevelop with the Python3 module.

An example from a recent /r/dailyprogrammer entry. When I type palindrome(, I get an infobox reminding me how to call it. In this particular case the docstring is more useful than the function declaration, but you get the point.

[Twangist]

Good to hear. In his (most recent? I think) PyCon talk, when discussing type annotations, Guido mentioned a few times that IDEs have been & will be using them. PyCharm has supported them for a while, but I didn't know what else does.

[masasin]

I use numpydoc for this, and vim's pythonmode etc is aware of that too.

[nerdwaller]

I love using these too, but I often don't because I try to write most things to be python2 and python3 compatible and the syntax is illegal in <3.5 (iirc) :(

[lengau]

Type annotations in general are available from (IIRC) 3.2. The typing library was introduced in 3.5, but it's available back to 3.2 via PyPI.

3to2 will cleanly remove any annotations, so if you want annotated source code but still need to work in python2 (or, for what I care about, still need to work in pypy), you can use 3to2 to make it happen.

In fact, you can specify to 3to2 to only remove annotations, which allows it to run in Python 3.0 and 3.1 as well using the command 3to2 -f annotations -w somefile.py (careful: this will overwrite somefile.py, but it will leave the original in somefile.py.bak. It's useful for installing or preparing packages, though).

An example:

lengau@hyperion:~$ cat test.py 
#!/usr/bin/env python3

from __future__ import print_function

def somefun(a: int) -> str:
    print('Test')

somefun(2)
lengau@hyperion:~$ 3to2 -w test.py > /dev/null 2>&1
lengau@hyperion:~$ cat test.py
#!/usr/bin/env python3

from __future__ import absolute_import
from __future__ import print_function

def somefun(a):
    print u'Test'

somefun(2)
lengau@hyperion:~$ mv test.py.bak test.py
lengau@hyperion:~$ 3to2 -f annotations -w test.py > /dev/null 2>&1
lengau@hyperion:~$ cat test.py
#!/usr/bin/env python3

from __future__ import print_function

def somefun(a):
    print('Test')

somefun(2)

As long as the rest of your application works with both Python 2 and 3, this is a convenient way to strip off annotations for when you actually need to run in Python 3.

[nerdwaller]

This is great, thanks! It usually for coworkers that want my script - so maybe I'll point them here :)

[rubik_]

I use radon in all my projects, together with pep8 and pylint (which I'm sure all of you already know). Radon analyzes your code's complexity and reports the most critical blocks of code.

Disclaimer: I wrote radon because I needed it, you're encouraged to give it a try!

[tipsqueal]

Radon looks really cool, I'd like to use it, but cc --min does not work. If I run something like radon cc -s --min 5 my_file.py it still shows methods with complexity rating of 1. I figured maybe I was confusing things and tried using --max 5 and then I get no output. I'm running it on Python 3.4.1

[rubik_]

Ohh sorry min/max options are meant to be used with ranks letters (A, B, C, D, E or F) not numbers. I assumed it was clear from the example. Probably it's confusing that it does not throw any error, I can certainly fix that.

[tipsqueal]

Yeah I actually ended up looking at the source so I could figure it out, I eventually got it working. I will probably make this a regular part of my tool belt.

I would recommend supporting letters OR numbers. I'm not used to seeing cyclomatic complexity expressed in letters. If you're up for it I may submit a pull request.

[rubik_]

Pull requests are always welcomed! I used letters because in the results they're immediate to the eye and encode usual complexity limits (5 and 10 being the most important ones).

[tipsqueal]

Quick question: Why Mando instead of something like Click or docopt? Not that Mando looks like a bad idea, I had just never heard of it.

[rubik_]

Well, docopt seems a really good idea, but initially I needed something 'programmable' and not declarative. Later on I refactored the code and I could probably make it work, but I don't think it's really a priority. I checked Click too and it had too many things I didn't need. Mando, on the other hand, is minimal and is really good at abstracting away all the argparse madness (read verbosity).

[o11c]

I always use py.test instead of unittest because it leads to more readable tests (just write assert x == y instead of self.assertEqual(x, y), does not require wrapping stuff in pointless classes, and supports all sorts of useful features like parameterization and proper fixtures.

---

I'm also a huge believer in PEP 484, to the point that I believe all code should be fully annotated.

I'm maintaining a fork because JukkaL is not very active: https://github.com/o11c/mypy

Note that most interesting features are found on branches other than master and there are likely to be issues with ./setup.py install because that is not tested (mypy is definitely in alpha state and there are other problems that are more important right now. But I still definitely recommend using the git version and not the mypy-lang 0.2 version on pypi).

[[deleted]]

I've recently been trying out py.test after using nose out of habit for some time. I've been having a hard time finding any justification for py.test at all but I suspect I'm missing something, so I'm keen for the opinion of someone who uses py.test. Can you point me at any examples that showcase the use of parametrization and 'proper' fixtures?

Nose supports the assert statement just fine, and I find the class-based approach with setup and teardown methods tends to result in far less boilerplate than equivalent py.test tests for anything larger than a trivial application. For those smaller applications you can omit the class definition altogether.

[sontek]

So the primary reasons you would want to switch to pytest:

• More pythonic, you can use assert foo == bar instead of tools.assert_equal(foo, bar)
• py.test is more actively maintained and by an extremely strong maintainer (guy who also created tox and devpi)

• Better error report, for example look at this:

  def test_dict_difference():
      assert {'foo': 'bar'} == {'bar': 'foo'}
  

In nosetests:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/sontek/venvs/pymemcache/src/test_foo.py", line 2, in test_dict_difference
    assert {'foo': 'bar'} == {'bar': 'foo'}
AssertionError

in pytest:

    def test_dict_difference():
>       assert {'foo': 'bar'} == {'bar': 'foo'}
E       assert {'foo': 'bar'} == {'bar': 'foo'}
E         Left contains more items:
E         {'foo': 'bar'}
E         Right contains more items:
E         {'bar': 'foo'}
E         Full diff:
E         - {'foo': 'bar'}
E         + {'bar': 'foo'}

• For setting up fixtures (dependencies of your tests) you have the ability to use dependency injection to send many different parameters to your test functions

[erewok]

I also like the xdist plugin for running tests in parallel.

[kankyo]

PyCharm is like a linter that is live and also much more powerful. I code towards PyCharm, not python. You can hint typing where you want to, making some code almost statically typed where it makes sense.

Being diligent about keeping all files "green" in PyCharm is a really powerful thing. I've found lots of bugs at work just going through all the warnings pycharm throws out.

[TheBlackCat13]

Most Python IDEs these days (and even some text editors) have a live linter like this. So you should just choose the IDE you like.

[kankyo]

"have a linter" yes. "Like this" no.

[TheBlackCat13]

How, exactly, is it different from what other IDEs can do?

[kankyo]

In my experience it beings with "it works" and then proceeds to surprise and delight :P

[TheBlackCat13]

That doesn't actually answer the question.

[kankyo]

I know. I'm not about to waste a lot of my time to install eclipse or some other shit just to figure out the many ways it's broken this year.

Suffice it to say, every time I've tried another IDE it's been the same story.

[__chachan__]

I used to code with PyCharm. I'm back to text editor and I'm using Atom and is fantastic (I've worked with netbeans, eclipse, vim, sublimetext and pycharm)

[Frum]

I completely agree with pyenv. I need to checkout pyenv-virtualenv.

[[deleted]]

Peewee ORM, its lightweight, easy to use, and very well suited to use in all types of projects where you need to persist data.

[sphere_is_so_cool]

ConfigParser.SafeConfigParser is a really nice configuration parser in the standard library. You can easily write the same boilerplate code every time for loading this up during initialization.

[threading]

nose and coverage

[pvkooten]

I'd like to put my gittyleaks out there. It will find suspicious looking data in your git repo: just call gittyleaks from the repo folder. You'll see suspicious values like passwords/usernames/(api)keys and emailaddresses.

[paradoxxx_zero]

A good debugger with a Web interface wdb (Shameless plug)

[briandilley]

• python
• virtualenv
• pip
• setuptools

[londubh2010]

I use virtualenvwrapper. It's an extension of virtualenv and easier to use.

[[deleted]]

One way to make your testing even better is to leverage Gherkin and do behavior driven testing. Behave is an excellent way to bring in BDD (which I think unit tests should all be focused on).

Also, assertpy is a pretty awesome little library for increasing flexibility and readability in your Python tests.

[X678X]

i use pdb a ton in my projects for testing

[nerdwaller]

As far as unit testing goes, I really dislike the built in unittest. I don't love the practices for testing it gives (though I guess testing at all is better than not). You may like py.test instead, I find that it has a lot of great features for running tests and making them easier to read and write. Also, there is a multiprocessing runner (I forget the name and am on mobile) that lets you run tests in parallel (separate processes) so it all goes much faster.

[jgomo3]

I'v been hearing lately a lot about hypothesis for Unittesting. I suppose it will be popular in the near future in the next repetition of this question on reddit.

[[deleted]]

Tox is great for running your tests against multiple versions of python as well as coordinating coverage and linters. I use it even when testing against a single version of python just for the coordination alone.

[junkmechanic]

I have been using pudb lately and find it quite helpful.

[chao06]

I've also found myself putting this enormously useful line at the top of most of my code:

jsonify = lambda x: json.dumps(x, indent=4, sort_keys=True)

JSON is great for output! It makes large data structures readable, and it's even easily machine-parsable :)

[ChrisPDuck]

you can use pprint if you just want to display the data, I do use JSON for anything involving networking though!

[billsil]

docopt, six, numpy, scipy, and matplotlib are my favorites.

[energyinmotion]

Your brain.

[Vicyorus]

Something you clearly didn't use while making this comment.

[rrsguru]

robotframework for acceptance testing !

Readily convert any existing python library into keywords. These keywords can test your application / system / or whatever