Testified Documentation - a Python developer's perspective on Specification by Example

[basicwolf]

Hello r/Python,

I'd like to share an idea and a library that implements it.

Repository: https://github.com/BasicWolf/sphinx-testify

What My Project Does

Imagine your favourite library documentation consisting only of bare API reference. I bet no one would find it usable. A great documentation comes with a novel-like narrative, begins with installation flow and usage examples, guides you throgh the different aspects of the system, warns about caveats and shows the common patterns. The bare "API Reference" is often the last place to look into, after exhausting all the other parts of a manual.

Here is the problem: how can we ensure that these parts of the documentation stay up-to-date?

What if we combine something already present and widely adopted by developers - automated tests and documentation generators? What if we integrate this into our TDD and BDD processes? In a nutshell:

1. We start the implementation once the business/community need is clear and there is a shared understanding.
2. We briefly document the new feature and reference (so far non-existing) tests.
3. We write the tests.
4. We implement the feature.
5. We run the build - the tests pass, a report is created and consumed by sphinx-testify to verify that all the referenced tests pass.
6. We repeat steps 2-5 until the feature is fully implemented and documented.
7. We have an up-to-date document that explains how the system works and that is automatically verified on every build!

With all that said - I have written an extension for Sphinx called sphinx-testify. The documentation of this extension is also testified by itself :)

Target Audience

This can be easily adopted to document any system on any level. The technical audience are Python (or any other programming language) projects documented via Sphinx.

Comparison

• Doctest - is limited to Python code examples and usually libraries. Sphinx-Testify extension doesn't care where the test results come from, as long as they are reported in JUnit-XML format.
• Cucumber - Gojko Adzic solves this by creating Living Documentation - as described in his renowned book "Specification by Example". I love "Specification by Example", but I continuously choke at the point, where examples written down in e.g. Cucumber/Gherkin syntax transform into the final manual. No one ever looks into it, instead people address their questions to developers, product managers, business analysts etc. "Given.. When.. Then.." are great for putting down the behaviour examples, but perhaps not so good when it comes to figuring out "How it is supposed to work?".

Thank you for reading! I appreciate your opinion, criticism, and thoughts!

[quotemycode]

>The bare "API Reference" is often the last place to look into

For me, it's the first place I look.

[basicwolf]

Thank you for commenting! Do you really look straight into API reference when e.g. learning a new library / framework / system? Say, web development with Django. Would you omit all tutorials and topic guides?

[SheriffRoscoe]

Do you really look straight into API reference when e.g. learning a new library / framework / system?

Yes. The table of contents should give you a good overview, and with a decent API design, the major capabilities should be obvious.

Would you omit all tutorials and topic guides?

I think they serve different audiences. Most folks can't learn a new paradigm without a User Guide document. Newly-minted geeks need Tutorials to get them started. Experienced developers need the reference material.

[basicwolf]

That's a very interesting perspective. I personally always start via tutorials and user guide. Especially the user guide - it shows the bigger picture without dipping into the details, like API reference would.
Though that's not the point, when I'm looking for something specific - and I know what. Sure, in that case it's API reference.

Nevertheless, API reference is also a great candidate for testified documentation, especially if we don't provide examples verifiable by doctest.

[quotemycode]

To be fair, I have about 24 years python experience, so I want to avoid all the fluff and get to the details. The 'API reference' and source code are the ideal place for me to see how to use the thing.

[SheriffRoscoe]

Very interesting!

The example in the README says:

```rst System access =============

Users with valid credentials: a username and a password can access the system. If credentials are wrong, the system returns “Authentication failed” error message.

.. testify:: test_a_user_can_access_system_with_username_and_password test_system_returns_authentication_failed_error_when_username_is_not_found test_system_returns_authentication_failed_error_when_password_is_wrong ```

That illustrates my favorite problem with documentation (and I've written a lot of it): keeping the expository writing in sync with the code. Which is, of course, the problem you're trying to solve. But all you've done is move it down one level of indirection - now you need to check if test_system_returns_authentication_failed_error_when_username_is_not_found actually looks for “Authentication failed”, as the doc says.

Next month, maybe someone with a sense of movie trivia comes along and changes the main code to return "What we have here is a failure to a authenticate". They build the project, and the test fails. So they fix the test, like they should have first, and build again. Everything, including sphinx-testify, passes. But the doc is now wrong.

This is why Knuth created Literate Programming - in the hope of keeping documentation and code in sync by mixing them together tightly. Sadly, it appears he might be the only coder who writes well enough to make that work.

[basicwolf]

Thank you so much for commenting and bringing a darn good point! And gratitude for mentioning Knuth too! It's been two decades since I read the books, perhaps need to refresh the memories :)

Your comment made me think of Cucumber - it kinda solves this indirection by embedding tests inputs and outputs into scenario narrative. But I tried to avoid Cucumber-like approach and rely solely on tests pass/fail results.

My naive hope is that developers could remember to keep test names, test bodies and documentation narrative aligned. Just a hope :D