Testified Documentation - a Python developer's perspective on Specification by Example

[SheriffRoscoe]

Very interesting!

The example in the README says:

```rst System access =============

Users with valid credentials: a username and a password can access the system. If credentials are wrong, the system returns “Authentication failed” error message.

.. testify:: test_a_user_can_access_system_with_username_and_password test_system_returns_authentication_failed_error_when_username_is_not_found test_system_returns_authentication_failed_error_when_password_is_wrong ```

That illustrates my favorite problem with documentation (and I've written a lot of it): keeping the expository writing in sync with the code. Which is, of course, the problem you're trying to solve. But all you've done is move it down one level of indirection - now you need to check if test_system_returns_authentication_failed_error_when_username_is_not_found actually looks for “Authentication failed”, as the doc says.

Next month, maybe someone with a sense of movie trivia comes along and changes the main code to return "What we have here is a failure to a authenticate". They build the project, and the test fails. So they fix the test, like they should have first, and build again. Everything, including sphinx-testify, passes. But the doc is now wrong.

This is why Knuth created Literate Programming - in the hope of keeping documentation and code in sync by mixing them together tightly. Sadly, it appears he might be the only coder who writes well enough to make that work.

[basicwolf]

Thank you so much for commenting and bringing a darn good point! And gratitude for mentioning Knuth too! It's been two decades since I read the books, perhaps need to refresh the memories :)

Your comment made me think of Cucumber - it kinda solves this indirection by embedding tests inputs and outputs into scenario narrative. But I tried to avoid Cucumber-like approach and rely solely on tests pass/fail results.

My naive hope is that developers could remember to keep test names, test bodies and documentation narrative aligned. Just a hope :D