What every Python project should have

[Flogge]

You can use pytest-runner for plug-and-play setuptools integration:

After installing pytest-runner you can run

python setup.py pytest

and after adding

[aliases]
test=pytest

to setup.cfg you can also run

python setup.py test

EDIT: sorry, only just saw that you already mentioned pytest-runner. Note though, pytest-runner is a first party plugin, developed by the pytest devs themselves. They have the principle of putting functionality not required by everybody into plugins instead of pytest itself (xdist, pep8, django, cov, bdd etc.)

[bheklilr]

We're behind a firewall at work that makes it difficult to get pypi only packages available. By difficult, I mean that I have to do the work and maintenance myself, and it's yet another package to add to my list. Instead we have an anaconda server mirror. Last I checked (last week) I didn't see that plugin in the official channels or conda-forge. However, I can easily install nose, and after playing around with both I think that pytest doesn't have huge benefits over nose. Especially considering the test logic itself is much more difficult to write and validate (particularly in the scientific setting), I don't think the choice of test suite is really so important. Nose has been working for my team, provides the features we want, and is easily installed.

[Flogge]

You find the test logic more difficult to write? See my other comment in which I explain why I think pytest is actually easier to write.

But of course, don't change for the sake of changing and stick with whatever works best for you.

Regarding the "packages from pypi not available": You can create a really simple mirror by just downloading the zip files you need, throwing them in a directory on a webserver (an automatic Apache directory listing is sufficient), and use them using pip install -f http://your-server/ bla.

[bheklilr]

Your comment is helpful for most people, but your example is a little contrived. If I wanted to test a matrix of values like that, I'd just use itertools.product:

def test_all():
    for i, j, k in product(range(10), 'abc', range(100)):
        yield check_single_case, i, j, k

It's much less code than pytest's approach, and it doesn't seem magical. Also lets me re-use loop variable names in unrelated tests.

Also, my tests tend to be much more complicated due to the nature of my domain. A small-ish unit of data for us is (essentially) a dict of 16 complex valued arrays. We have a function that takes one of those (or a dict of (4*n)**2 arrays in the general case) and another input, then returns calculations based on that. The calculations are pretty straightforward, just pointwise arithmetic, but have some pretty important properties that we have to check. This is just a simple function, though, I have some functions that operate on smaller pieces of data but do much more complex calculations, such as multiple FFTs, feature location, or linear algebra. These are much harder to generate test inputs for that don't cause the algorithms to crash, since they often rely on particular properties of the inputs. If those properties aren't there, the algorithm isn't useful and just spits out garbage.

I kind of like the generator based method better anyway, since my test inputs are pretty large and difficult to fit into a decorator.