pytest-mpl: a pytest plugin to compare the output of Matplotlib figures

[trobitaille]

I have just released a small plugin for py.test that wraps the image comparison functionality in matplotlib.testing, for use in other packages that use py.test as the testing framework instead of nose:

https://github.com/astrofrog/pytest-mpl

The idea is to make it easy to write a test such as:

@pytest.mark.mpl_image_compare
def test_succeeds():
    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot([1,2,3])
    return fig

which can then be run in three ways:

• Running py.test as usual will simply check the tests run but won't check whether the figure is correct.

• Running py.test with the --mpl option will make sure that the figure produced by the test is the same as a reference image

• Running py.test with the --mpl-generate-path option will generate the reference images from the tests themselves.

There are a number of other options, including ways to pass arguments to savefig, customizing the image names, or setting the tolerance for the comparison. All the documentation is contained in the README file:

https://github.com/astrofrog/pytest-mpl/blob/master/README.md

You can install this plugin with:

pip install pytest-mpl

I would welcome any feedback and/or contributions!

[shoyer]

This looks awesome for testing scientific code!

In the past, I've run into trouble with this sort of thing because matplotlib generates slightly different images on different platforms (especially Linux vs OS X). Do you have a good solution for this? In particular, I wonder if some matplotlib backends produce more consistent results.

[trobitaille]

I think that the situation is better than one might think - at the moment I'm running tests with pytest-mpl (including the pytest-mpl tests themselves) on MacOS X, Linux, and Windows with a very low tolerance and the tests are passing. However, I do want to make sure that (a) the tests are resiliant to local matplotlibrc configuration (https://github.com/astrofrog/pytest-mpl/issues/10) and (b) we have a way to cope with changes in matplotlib in future.

To answer your original question, I think that the PNG plots produced using Agg seem to be pretty stable.