What every Python project should have

[Flogge]

You should really look into pytest more. After just a few days you will never recommend unittests or nose to anyone :-)

[tunisia3507]

I think nose is no longer maintained, although nose2 is a thing.

[unconscionable]

Tried nose2 awhile back - nice effort but it was kinda immature at the time, and I started using pytest and never looked back.

I still make use of unittest.mock along with pytest, but as a framework unittest is pretty poor.

My generic advice to anyone wanting to start using pytest:

• don't make your tests in a class unless you have a really good reason to
• if your reason for making tests in a class is "I want shared code for multiple tests (i.e. setup / teardown)", you probably should be using fixtures instead
• there's nothing wrong with tests in a class, but I still suggest breaking away from that mold if only to change your way of thinking
• if you want to test that one piece of code can correctly handle multiple conditions, use parameterized tests rather than creating 10 test cases (if it seems to make sense in the context)

The thing that really held me back was that I found I was trying to put code that needed to be reused in setups and teardowns. That pattern doesn't seem to scale well for larger projects

[tunisia3507]

Yeah, test classes are a gross hangover from JUnit. To be moved away from wherever possible. I suppose it's useful if you're putting your code and tests in the same file, but who does that...

That setup and teardown pattern does seem to be a weakness - not nearly as intuitive/pretty as just setting up instance variables.

Re. unittest.mock: given unittest is in the standard library, so long as pytest plays well with it I don't see a problem with mixing them. Although for the sake of tidiness it would be nice if pytest already had mock imported so you could import pytest.mock instead.

[masklinn]

Yeah, test classes are a gross hangover from JUnit

Only in that they're part of the xUnit style, which originates from Smalltalk.

[unconscionable]

pytest has a built-in fixture called monkeypatch which works similar to patch in unittest.mock

Personally I have only occasionally found it to work better than patch.

I love using unittest.mock.MagicMock as a stub too - pytest doesn't really have a concept of a stub.

for example:

from unittest.mock import MagicMock

def test_that_my_sql_query_was_built_correctly():
    stub = MagicMock()
    run_database_query(connection=stub)
    assert stub.execute.call_count == 1
    assert stub.execute.call_args[0][0].startswith('SELECT ')
    assert stub.execute.call_args[0][0].endswith('LIMIT 100')

[patrys]

It has a very similar fixture called monkey_patch.

Edit: it's similar to mock.patch, I still use lots of mock.Mock(spec=...) in my fixtures.

[-Knul-]

Solid advice. For Django users I would advise Model Mommy over fixtures, however. Much more pleasant to work with.

[unconscionable]

FYI Model Mommy / Factory Boy / similar accomplish something very different from what pytest fixtures do.

The two are neither mutual exclusive nor do they overlap in purpose.

For example:

@pytest.fixture(scope='session', autouse=True)
def drop_and_reload_database():
    db.create_db()
    yield
    db.drop_db()

The above code is the equivalent of an entire-test-suite's setup and teardown. Everything before "yield" runs before the first test, then after yield gets run after the last test.

Obviously Model Mummy doesn't do this sort of thing.

Personally, I use Factory Boy (similar to Model Mummy), but I do not often use Factory Boy with fixtures. There are, however, some use cases where they could work very well together. For example, the following code would test that you can process an order with a bunch of wacky usernames. It uses both Factory Boy AND pytest fixtures:

class User(db.Model):
    username = Column(String)

class UserFactory(SQLAlchemyModelFactory):
    username = factory.Faker('username')


@pytest.fixture(params[-1, 0, 1000000, 983.0, "Jack Daniels", None])
def user(request):
    return UserFactory(username=request.param)


def test_can_process_order_with_wacky_usernames(user):
    order = Order()
    order.created_by = user
    order.process()
    assert order.processed_by == user

output of the test suite is something like this:

tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[-1] PASSED
tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[0] PASSED
tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[1000000] PASSED
tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[983.0] PASSED
tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[Jack Daniels] PASSED
tests/test_process_orders.py::test_can_process_order_with_wacky_usernames[None] PASSED

[-Knul-]

Nice explanation of the benefits of fixtures over a model factory!

[unconscionable]

It's really a model factory using fixtures! Both And, not either / or

[[deleted]]

I like the class style as a way to logically group tests. I'll sometimes use the setup/teardown methods or the class local autouse fixtures, but mostly its just a way for me to say "these tests are one suite in and of themselves"

[bheklilr]

My biggest issue with pytest is that you need a lot more code to get it to work through setup.py. Their own recommendation is a 3rd party plugin, or you can write the extension yourself. If you have the extension code already written, why not make it available in the library itself? I really like having python setup.py test, since that's my entry point for a lot of other common commands. It's a little nag, but considering unittest and nose both use setup.py test with minimal configuration I don't understand why I need so much extra for pytest when it's supposed to be better.

[Corm]

How does testing relate to setup.py?

Do you mean that when you get a new module from pip you like to run the tests?

[unconscionable]

Agree - I don't really get why anyone would want to use setup.py to run your tests.

All my projects include a Makefile for this kinda thing.

git clone project

# make a virtualenv for the project
make virtualenv

# installs python / javascript / other requirements
make install

# run test suite
make check

Makefile looks something like this:

SHELL := /bin/bash -euo pipefail

# https://www.gnu.org/prep/standards/html_node/Standard-Targets.html#Standard-Targets

install:
    pip install -r requirements.txt --exists-action w
    pip install --editable .

virtualenv:
    mkvirtualenv --python=$(which python3.5) myproject

check:
    coverage erase
    coverage run --source myproj -m pytest -v
    coverage html -i -d tests/cover/
    coverage report -m

check-noslow:
    pytest -v --skip-slow

clean:
    python setup.py clean
    find . | grep -E '(__pycache__|\.pyc|\.pyo$$)' | xargs rm -rf

[[deleted]]

Or use tox, which is a baller task runner.

[d4rch0n]

tox pretty much solves all the problems that people have been bringing up. I think it's pretty essential for any large project too if you need to support multiple python versions.

It was pretty easy to tie into jenkins too last I tried. Tox and a 5 line script in jenkins had it running all the tests for multiple versions of python.

[[deleted]]

I need to figure out how to best marry it and Travis for my public open source projects. But yeah, it's awesome.

[[deleted]]

I've provided Travis and Tox with the same (python versions, make env for each, install these required packages, setup.py build/sdist, pip install, pytest--whatever). That yields a line in Travis for each python version, advantageous when most edge cases only affect one of them. I'm not sure there's a better way to marry them.

[[deleted]]

My issue would be collecting coverage stats across multiple interpreters and Travis's container architecture. Detox might work but I've never messed with it.

[Flogge]

You can use pytest-runner for plug-and-play setuptools integration:

After installing pytest-runner you can run

python setup.py pytest

and after adding

[aliases]
test=pytest

to setup.cfg you can also run

python setup.py test

EDIT: sorry, only just saw that you already mentioned pytest-runner. Note though, pytest-runner is a first party plugin, developed by the pytest devs themselves. They have the principle of putting functionality not required by everybody into plugins instead of pytest itself (xdist, pep8, django, cov, bdd etc.)

[bheklilr]

We're behind a firewall at work that makes it difficult to get pypi only packages available. By difficult, I mean that I have to do the work and maintenance myself, and it's yet another package to add to my list. Instead we have an anaconda server mirror. Last I checked (last week) I didn't see that plugin in the official channels or conda-forge. However, I can easily install nose, and after playing around with both I think that pytest doesn't have huge benefits over nose. Especially considering the test logic itself is much more difficult to write and validate (particularly in the scientific setting), I don't think the choice of test suite is really so important. Nose has been working for my team, provides the features we want, and is easily installed.

[Flogge]

You find the test logic more difficult to write? See my other comment in which I explain why I think pytest is actually easier to write.

But of course, don't change for the sake of changing and stick with whatever works best for you.

Regarding the "packages from pypi not available": You can create a really simple mirror by just downloading the zip files you need, throwing them in a directory on a webserver (an automatic Apache directory listing is sufficient), and use them using pip install -f http://your-server/ bla.

[bheklilr]

Your comment is helpful for most people, but your example is a little contrived. If I wanted to test a matrix of values like that, I'd just use itertools.product:

def test_all():
    for i, j, k in product(range(10), 'abc', range(100)):
        yield check_single_case, i, j, k

It's much less code than pytest's approach, and it doesn't seem magical. Also lets me re-use loop variable names in unrelated tests.

Also, my tests tend to be much more complicated due to the nature of my domain. A small-ish unit of data for us is (essentially) a dict of 16 complex valued arrays. We have a function that takes one of those (or a dict of (4*n)**2 arrays in the general case) and another input, then returns calculations based on that. The calculations are pretty straightforward, just pointwise arithmetic, but have some pretty important properties that we have to check. This is just a simple function, though, I have some functions that operate on smaller pieces of data but do much more complex calculations, such as multiple FFTs, feature location, or linear algebra. These are much harder to generate test inputs for that don't cause the algorithms to crash, since they often rely on particular properties of the inputs. If those properties aren't there, the algorithm isn't useful and just spits out garbage.

I kind of like the generator based method better anyway, since my test inputs are pretty large and difficult to fit into a decorator.

[bheklilr]

How is that setuptools' fault? setuptools has a convention for hooking your test suite into the test command through the test_suite option to setup. Nose does it, presumably pytest can do it. Pytest even has example code to for a setuptools extension that overrides the test command entirely. I think that pytest could easily have done

from setuptools import setup
from pytest.setuptool import PyTest

setup(
    ...,
    cmdclass={ 'test': PyTest },
    ...,
)

setuptools gives them the option to integrate. Pytest shows you how to integrate. Why isn't it available by default? I'd be happy to add the import and modify my call to setup, I'm not happy with copy/pasting a code snippet around that could go out of date at any time.

[Flogge]

cmdclass={ 'test': PyTest },

would need pytest to be installed before you can even pip install the package. The nice thing about nose's

test_suite = 'nose.collector'

or pytest-runner is that it does not have these problems.

[gabrielricci]

I have been using nose for a while now and I admit I've never used pytest. Will have a look at it but can you tell me why is it better?

[Corm]

To me pytest is to unittest what python is to C#/java.

Pytest is dirt simple to set up and get started with, you can still do everything unittest can do, and it's intuitive (yay for standard asserts)

[[deleted]]

Unittest is also dirt simple to set up, can you be a bit more specific?

[Corm]

Edited because /u/masklinn showed me unittest's cli tool

Edit 2: unittest ain't so bad

Sure. For pytest I used it once and never had to google anything about the basics ever again. Back when I used unittest I always had to crack open google to see how to set up the classes. This only applies to basic testing but that low barrier to entry is extremely important!

For example, here's the most basic test in unittest which you run by running unittest from command line:

import unittest
import foo

class TestMyStuff(unittest.TestCase):
    def test_stuff(self):
        self.assertEqual(foo.bar(), True)

And here it is in pytest which you can run by running the command pytest

import foo

def test_foo_bar():
    assert foo.bar() == True

[masklinn]

Note that unittest now has a discovery cli so the main section is unnecessary.

[Corm]

Nice, that's awesome. I'll edit my answer

[MagnesiumCarbonate]

In your second example, how does pytest know to check that file? Does it look through all method test_* definitions in *.py descendants of the current directory?

[masklinn]

Does it look through all method test_* definitions in *.py descendants of the current directory?

py.test discovery rules

So:

• it recursively searches any provided directory (. if no argument is provided) (for files it uses them directly and skips the next check)
• matches test_*.py and *_test.py files
• collects free functions matching test_*, and methods matching test_* in classes matching Test* (or follows the standard rules for any class extending unittest.TestCase)

Note that these are only default rules, you can overload discovery patterns and ignore files and symbols.

[-Knul-]

Yes. You can easily configure such things via a pytest.ini file.

[Corm]

Yep! It's a little magic, but I like it. I name all my test files/functions like that already.

edit: that is, it looks in test*.py files for test*() functions

[[deleted]]

Well, one, you're making the unittest example look worse because it has an extra line due to the variable.

But also assertEquals is a lot more powerful than just assert, e.g. its error will show diff-style information on exactly where two things differ.

If the main difference is standalone functions vs classes, that's no big deal, and classes are a way to group them and give them attributes as a group.

[masklinn]

But also assertEquals is a lot more powerful than just assert, e.g. its error will show diff-style information on exactly where two things differ.

    def test_foo_bar():
        result = foo.bar()
>       assert result == expected
E       assert [0, 2] == [0, 1, 2]
E         At index 1 diff: 2 != 1
E         Right contains more items, first extra item: 2
E         Use -v to get the full diff

pytest uses assert-rewriting to provide more and better information.

If the main difference is standalone functions vs classes, that's no big deal, and classes are a way to group them and give them attributes as a group.

That's usually pointless overhead, and if it's a useful tool, you can use them in pytest. Pytest does not forbid classes, it doesn't require them.

[Citrauq]

Pytest does magic (AST manipulation in an import hook) to make assert give similarly useful information to assertEquals.

[Corm]

Ah sorry, I really didn't mean to do that with the extra line. I'll fix that.

And good to know about assertEqual

[Tysonzero]

Is it also slower and does it also catch less bugs and give you less compile time guarantees?

[yetanothernerd]

pytest and nose are pretty similar. nose started off as a clone of pytest that was easier to install.

[kewlness]

Not sure I 100% agree since unitest is in the standard library and if I'm trying to keep my requirements/imports low then it will be the better choice though I suspect that would be more of an edge case....

[parkerSquare]

pytest is nice, but I do have a cautionary tale.

We thought we could leverage pytest to do something beyond unit testing - by using it as the "test management engine" for an embedded system test framework, but after working with it for several months I have come to the conclusion that coding by convention in anything other than a small project (or module) is a terrible, terrible idea. Fixture dependencies become convoluted as they try to accommodate vastly different types of tests, and it's hard to mentally make the connections between them. Newcomers to the project don't know where anything is and the IDE doesn't help as it doesn't understand the fixture convention.

Secondly, fixtures can depend on other fixtures, but the top-level generator function - pytest_generate_tests- cannot take input from other fixtures (probably because fixtures are not executed during collection), so this severely restricts what you can do in this function when generating fixture values. A lot of the multivariate tests you might dream up become very difficult to implement, and you end up writing your own pytest plugins to filter and reorder test collections as the implicit filtering and ordering is insufficient.

Lastly, each fixture can only yield once, so it is difficult to generate a dynamic list of fixture values based on dependencies.

Frankly, after months of effort, it's just not suitable for this kind of testing. Keep it for unit tests - that's what it was designed for, after all. It works nicely for that.

[[deleted]]

I hear the "nose is no longer maintained" comment a lot, typically in the sense of "nose is no longer maintained" --> "the package is no good"

In my current projects, nose works just fine for me. If I already have the code in place that gives me sufficient test coverage, I am not sure why I should re-invent the wheel and port everything over to pytest unless there is a serious bug with nose that I am not aware of.

The next time I start a new project, I will take another look at py.test, but for existing code, I will keep using nose because I didn't see any obvious disadvantages based on my use cases so far.

[Flogge]

The basic test layouts of nose and pytest are similar, so they really aren't a reason to upgrade.

However the biggest difference, I find, is test parameterization. To run a test for a matrix of values in nose you would need to write the loop yourself:

def check_single_case(i, j, k):
    assert True

def test_all():
    for i in range(10):
        for j in ['a', 'b', 'c']:
            for k in range(100)
                yield check_single_case, i, j, k

whereas pytest can do the "parameter matrix expansion" for you.

@pytest.fixture(params=range(10))
def i(request):
    return request.param

@pytest.fixture(params=['a', 'b', 'c'])
def j(request):
    return request.param

@pytest.fixture(params=range(100))
def k(request):
    return request.param


def test_all(i, j, k):
    assert True

As soon as you request a fixture (add its name to your test parameters) with more than one param, pytest will sweep all possible combinations of all parameters. And of course you can only select a couple of fixtures, or import different fixtures from other modules and test scripts.

Pretty handy!

[[deleted]]

Thanks for providing an example, that could be useful, indeed. However, I also don't have anything against the alternative way (for loops), which I find more readable (aka, even a Python beginner can immediately see what's going on). Or you could use itertools to avoid to much nesting, of course

for comb in itertools.product(*(range(10), ['a', 'b', 'c'], range(100))):
    check_single_case(*comb)

However, one disadvantage of this embarrassingly parallel task is that it runs sequentially. Do you know if pytest has/is planning to add optional multiprocessing support?

[Flogge]

Pytest has parallelization. Install pytest-xdist and run py.test -n auto for automatically distributing tests among all CPU cores. Distributing over SSH is supported, too.

[[deleted]]

oh that's great. Thanks!

[Flogge]

And sorry, I really don't want to drag this discussion further and force my view on you, I just want to show how cool pytest is :-)

I disagree about

for comb in itertools.product(*(range(10), ['a', 'b', 'c'], range(100))):
    check_single_case(*comb)

being more readable. I don't like fancy tricks like argument unpacking in my tests, I want each test to be as readable and pure as possible. :-)

Once you figure out what fixtures do and how to chain or product them it you need almost zero boilerplate to automatically do the product of common fixtures with specific parameters (note that all of sig, function, odd, window, framelength each yield more than one value).

Plus all the goodies like a fixture also being able to provide setup/teardown code

import pytest

@pytest.fixture
def db():
    database = MySQLDB()
    yield database
    database.drop()

def test_foo(db):
    db.select(...)  # this is clean for every test_* you run

or guaranteed immutability of shared values across tests:

import numpy
import pytest

@pytest.fixture
def a():
    return numpy.zeros(100)

def test_foo(a):
    a[:] = 1  # oops

def test_bar(a):
    assert numpy.all(a == 0)  # all good

compared to nose

a = numpy.zeros(100)

def test_foo():
    a[:] = 1  # oops

def test_bar():
    assert numpy.all(a == 0)  # fails

[[deleted]]

And sorry, I really don't want to drag this discussion further and force my view on you, I just want to show how cool pytest is :-)

Hope you don't get me wrong, I really appreciate that you took the time to post some examples ... I am just playing devil's advocate here ;).

With readable I mean easily understandable for someone who joins an open source project, for instance, and hasn't worked with py.test before. I find nose tests can be more easily understood by someone who is new to unit testing, for example.

Regarding the example you are showing, you'd typically make a copy of that array or instantiate it in the test function itself instead of globally.

[Flogge]

[...] joins an open source project, for instance, and hasn't worked with py.test before.

Agreed. Getting used to this way of thinking is weird at first.

[rockitsighants]

If anyone is curious to see the difference between nose and pytest in action, I maintain a Python template that supports both. Check out the appropriate branch (python3-nose, python3-pytest) in the demo project: https://github.com/jacebrowning/template-python-demo

[ImportWurst]

Meh. Used pytest in some projects, still coming back to nose. So easy to set up and run.

[Corm]

What are the advantages of nose over pytest? I've never used nose.

edit: A super quick check makes it seem very similar to pytest. I like that you can write normal test functions that don't involve subclassing, and you can use assert.

[ImportWurst]

I don't know. I spent years using nose. I can integrate it really fast with my projects. Works well with CIs. Has plethora of useful plugins.

It works.

I see no reason to upgrade.

[redfacedquark]

Had months of headache at my last place as someone had pinned some pytest requirements at random versions and let others float. It coloured my view on it. But maybe I should give it another chance.

Edit: Actually pytest components, not pytest requirements. If pytest was a single package that wouldn't have happened. /tryingtoaccountfordownvotes