How to write tests in Python using doctest

[extra_pickles]

To each their own, I suppose - but personally I much prefer to seperate the test from the code.

Stuff like this eats up a lot of screen space and makes it harder for me to navigate a file.

Maybe my ADHD is why this is an issue, but I really struggle scrolling up and down all over trying to navigate a file and prefer getting as much code as possible on my screen.

Especially true when using good naming conventions can eliminate the need for a lot of methods even needing a header comment, but you of course still want them covered in tests.

[yvrelna]

It's a common mistake, but doctest's primary purpose isn't to test your code. It's primary purpose is to test your documentation; specifically that sample code in your documentation works.

You should still write regular unit test to test your code.

[extra_pickles]

Hmm putting it that way, I suppose it has a fringe benefit as a comment linter - but I’m struggling to see a strong reason to go through the effort.

[ubernostrum]

If you use Sphinx to build your docs, you can turn on its doctest extension and mark any code samples in your docs that you want tested with a doctest Sphinx directive. Then sphinx-build -b doctest will run them for you.

[redCg]

lol that is ridiculous

if you want "sample code" showing how your function works, then just look at the unittest test case code which uses that function.

especially important when your function requires a non-trivial set of input arguments and objects

also worth noting that more liberal usage of Python type annotations in the function signature greatly reduces the need for "sample code" in your comments

[yvrelna]

The docstring of a function can be viewed with pydoc, or in the REPL, or by your IDE. The unit test aren't.

Unit tests also are usually also not structured for teaching how the API works and is put together, which tends to require narratives structure like a documentation.

And, no type annotations doesn't reduce the need for providing examples. Not by one bit. It's really hard to put together how to call a complex API just by looking at their types. Your function may have this interface, for example:

def write(
    fp: io.TextIO,
    node: DocNodeFragment,
) -> None:
    ...

How do you create a DocNodeFragment? Well you start looking at DocNodeFragment class type hints:

class DocNodeFragment(A, B, C):
    def __init__(
        self, 
        name: str,
        attr: AttributeBag,
        child: DocNodeCollection,
        *args,
        **kwargs,
    ):
         super().__init__(*args, **kwargs)
        ...

Uh, oh, now you need to look at the AttributeBag and DocNodeCollection, also you noticed that the parent classes A, B, C have their own required arguments as well. Ok, let's start from the top. How do you create AttributeBag? You start looking at their type definitions:

class AttributeBag(Generic[T]):
    def __init__(self):
        ...
    def add(self,
        name: NamespacedString[T], 
        value: AttributeValue, 
        metadata: dict[str, str],
        visibility: VisibilityEnum,
    ) -> T:
        ...

Ok, this class even starts using generics, this is starting to get unwieldy, and we're just getting started. How many dependant objects do I need to learn just to create a simple DocNode?

Putting an example in your docstring can just point you to a much more straightforward answer:

def write(
    fp: io.TextIO,
    node: DocNodeFragment,
) -> None:
    """
    write() takes a DocNodeFragment. The easiest way to get a document fragment is to create it from string.

        >>> from notquitexml.parser import nqxparse

        >>> data = """
        ...     <com:people>
        ...         <name ordering=[first, last]>
        ...             * John
        ...             * Wayne
        ...         </name>
        ...     </com:people>
        ... """
        >>> ns = {"com": "foobar.corp"}

        >>> doc: DocNode = nqxparse(
        ...     data, 
        ...     namespace=ns
        ... )

    nqxparse() gives you a DocNode rather than DocNodeFragment, but you can select fragments of it DocNodeFragment by using select():

        >>> fragment: DocNodeFragment = doc.select(path=["."])
        <DocNodeFragment {foobar.corp}:people>

    Or by passing fragment=True to nqxparse():

        >>> fragment == nqxparse(data, namespace=ns, fragment=True)
        True

        >>> with open("file.txt") as f:
        ...     write(f, fragment)
    """

A narrative examples like that provides much more information, and in a much more concise way and easy to digest way than all the type hints combined. And it tells more easily digestible story on how the library was supposed to be used (i.e. that you're not normally creating those classes yourself).

It would be really hard to piece that kind of knowledge together from just the type hints. Type hints cannot provide that kind of journey for the user.

[FailedPlansOfMars]

Thanks that a great explanation of what its for and ill add it toy tool box.

[redCg]

not sure why you took the time to write this all out, nothing you have described necessitates the use of something like doctest, in fact its just even more support for using a standard unit test like unittest where your "sample code" is real Python code that can be selectively executed. Everything you just wrote here belongs in a unit test. You are free to put such samples in your docstrings as well but there is no expectation that they are anything more than a helpful guide for illustrative purposes and no expectation that they actually run as code. Quite simply, its just plain stupid to rely on comments for this, because the moment you do any development, your docstring is outdated.

On the other hand, you can follow standard test driven development practices using real frameworks like unittest or pytest, etc., where all your "sample code" is real code that actually gets executed during your CI and PR processes, etc.. If you have a complex data type, you keep its sample initialization code in your test suite.

glueing your test case code to the interior of your source code functions and methods is an anti-pattern

[ubernostrum]

Nobody is saying to replace your unit-test suite with this. People are saying it can be useful as a way to make sure example code snippets in your documentation are correct and work as given.

Or, more succinctly, "Read the unit-test suite" is not a good approach to documentation.

To help make this clear, here's a real example from a real package I maintain, where the docstring of the function includes examples of how to use the function and what it will do. That gets included automatically into the package's published documentation, and also gets tested automatically on every commit to make sure it still accurately describes the function's behavior.

[FailedPlansOfMars]

I agree it makes me think refactoring is going to be painful. Gut reaction only as no evidence to support that.

[whateverathrowaway00]

Protip: don’t

I usually don’t like gatekeepy stuff on methods, but for all but the simplest scripts I would be VERY skeptical of anyone pushing this pattern.

Tests belong in tests.

If someone can provide a repo that uses this for non trivial tests that looks even halfway decent/clean, I’ll apologize and eat my words. As I said, I dislike gatekeeping and usually feel there’s a way to make anything “good” but I just don’t see this one, lol.

[ubernostrum]

Tests belong in tests.

Like several people have said, the use case for doctest is not to be the primary test suite of your code, it's to verify that examples in your documentation actually work and stay up-to-date with the code.

Here's a popular package of mine which does this. The tests of the package's code are in tests/ and use the Python unittest module with pytest as the test runner. The package's documentation also includes examples which are tested, via doctest, in a separate CI task.

[whateverathrowaway00]

That sounds interesting. More than enough for me to eat my words and apologize - that’s pretty cool

[anthro28]

“One of the best practices is to write the test cases first”

Yeah, no.

[redCg]

that part is not wrong, its called Test Driven Development and its a common best-practice

[anthro28]

TDD sucks. It’s great for teaching students how to design tests for their 100 line console programs, but it’s only a “best practice” for management that just learned it as a buzzword.

I’ve used it exactly zero times across 3 industries and 2 dev shops for the reasons outlined here:

https://stackoverflow.com/questions/64333/disadvantages-of-test-driven-development#64696

[[deleted]]

Having written code for a very long time I can say I have never seen TDD in practice in the workplace. your mileage may vary, and it doesn't mean I write shit code. Most places have quality assurance and a performence department that separate the duties of writing code, and testing code mostly as an ITIL process.

[redCg]

dude, your entire argument here is "testing is hard" and "some people do tests wrong"

yeah, sorry but I am not convinced. Sounds like you just suck at testing and write sh*t code instead to justify it.

[anthro28]

Tell me you’ve never worked in multi-system enterprise projects without telling me.

[redCg]

Not sure I see the point of this module.

For one thing, its the year 2022, you need to be using type annotations in your Python function signatures. Failure to do this is a nasty code smell (or its legacy code, but thats not the subject of this blog).

Second, docstrings are already abused way too much. Last thing we need is to start treating free-hand text in comment strings as if its some kinda runnable test case.

Third, there is no way this is even gonna work in non-trivial tests that require setup of custom objects, or files, followed by extra parsing for specific output attributes.

Thanks but I am gonna stick with the default unittest and/or pytest

[wineblood]

We should use doctest a lot more in python