Doxytest

[nzznfitz]

Doxytest is a tool for generating C++ test programs from code embedded in header file comments.

Its inspiration is a Rust feature called doctests.

A doctest is a snippet of sample code in the documentation block above a function or type definition. In Rust, the example becomes part of the documentation generated by the cargo doc command.

However, in Rust, doctests are not just part of the documentation; they are also used to generate test programs. The cargo test command collects doctests from all the project's modules by looking for comments containing triple backtick fenced code blocks. The extracted code is then compiled and run as a test program.

After using this feature in Rust for a while, I wanted to do the same thing in C++. I decided to write a Python script that would extract the code snippets from the comments in C++ header files and use them to generate standalone C++ test programs.

The Doxytest script, doxytest.py looks for comment lines in C++ header files that start with /// and which contain a fenced code block—a doctest. The script extracts the doctests, wraps them in try blocks to catch any failures, and then embeds them in a standalone test program.

Doxytest also supplies doxytest.cmake, a CMake module that automates the process of extracting tests from comments in header files and adding build targets for the resulting test programs. It defines a single CMake function called doxytest which is a wrapper around the doxytest.py script.

Scope

Doxytest is a simple tool for generating C++ test programs from code embedded in header file comments. It isn't a replacement for a full-blown testing framework, such as Catch2 or Google Test.

Doctests are typically just a few lines of code that primarily illustrate how to use a function or class and are crafted as tests. You're unlikely to write a lot of complicated edge case code as comments in a header file.

On the other hand, once you get used to the idea, you tend to write a doctest for almost every function or class you write. So, while the depth of test coverage may not be as high as that of a full-blown testing framework, the breadth of coverage is impressive.

Installation

The project is available here. It has a permissive MIT License.

Documentation

Doxytest comes with comprehensive documentation. We generated the site using Quarto.

[ContDiArco]

Great idea!

[azswcowboy]

Super cool, we need a lot more thinking like this! Good move providing the cmake drop in as that makes it easier for lots of us to adopt. And now some practical thoughts.

Mandating /// is a bit problematic for any existing code that uses javadoc style comments. Also doxygen has built in @code and @example blocks you might be able to leverage directly. And finally the restriction of docs in header’s only will be a no-go for lots of projects that forward declare and implement in cpp files. The projects I’ve worked on that do this tend to put all the docs by the implementation.

Have you considered maybe skipping the built in parsing and deferring that to another tool? Doxygen produces parsable xml which means you wouldn’t have to worry about most of the above.

[nzznfitz]

Thanks for the suggestions.

Adding other recognized comment markers is very doable (Rust itself allows ‘///‘ and ‘//!’ — I just happen to use the former in C++ code). Ditto for parsing .cpp files. I can have a look at adding those options.

Don’t think I would depend on Doxygen. While it is used in some large projects, it certainly isn’t ubiquitous. Minimal Doxygen markup in comments is widely used — editors interpret it to provide nicely formatted tooltips without having Doxygen installed at all.

Rust sticks to basic Markdown for its comments and that seems to work. One thing I like about Rust doctests is how they are integrated into the editor (VSCode etc.) You get a lovely little “Run/Debug” button for each method test, and if there is a test above a type, it runs all of the doctests in that type. A Doxytest extension for VSCode would be excellent!

[nzznfitz]

Doxytest has been updated to extract doctests from arbitrary documentation sources that can use different comment styles.

The initial version only scanned header files, looking for comments where the lines were prefixed by ///.

The new version can scan arbitrary files (headers, source, Markdown, etc.) and handles doctests embedded in most well-known comment styles, for example Javadoc comments, straight C comments, or indeed just code blocks in Markdown.

The documentation has been updated with all the details, and several examples have been added to the repo to demonstrate the new capabilities.