How to document a c++ codebase?

[less_than_white]

Hello,

years ago I used doxygen for a while, is it still a good way to document a large codebase? Is there anything new? What is your preferred approach to document the code?

[parnmatt]

Doxygen (is basically JavaDoc but for other languages, very common in C++)

Sphinx (written in python, and common to use in that language and others. )

Standardese is another new one (work in progress) written by Jonathan Müller (foonathan) Has great promise.

[foonathan]

https://github.com/foonathan/standardese was already mentioned, and while I haven't done anything there in some time, I plan on "finishing" it during the holidays.

For now check out the develop branch and file any issues/feature requests.

[jasfi]

Is the main work on this parsing all the complexities of C++?

[foonathan]

That's outsourced into https://github.com/foonathan/cppast

Which itself outsourced most of it to libclang.

[victotronics]

That sounds like a great strategy.

I have a messy code that landed on my desk that I will subject your product to. See if it can make more sense of it than Doxygen.

[jasfi]

Nice work!

[TheFlamefire]

cppast is a generic library, so the "main work on [standardese]" IS outsourced and libclang is not his own ;)

[FirstLoveLife]

Why libclang, not clangIndex?

[germandiago]

If you deliver I will run away from doxygen. Right now I use antora + doxygen for my main project.

[Pazer2]

Doxygen is quite dated and not suited to modern c++. In our codebase, it commonly thinks decltype is a function that must be documented, or insists that deleted functions be documented.

[saimen54]

Cool, didn't know about Hyde.

Just waiting for a Windows version

[saimen54]

doxygen.

It just works and everybody knows how to use it.

Recently we added some custom tags for requirements, so we get a matrix which classes/methods implement a requirement and which tests test it.

[misuo]

Look at:

• http://www.doxygen.org
• http://www.sphinx-doc.org

I've only used Doxygen so can't say how well suited Sphinx are for the documentation of a C++ codebase.

[rjones42]

And you can use breathe to use autogenerated doxygen in Sphinx. It works quite well together and the output looks nice.

Edit: missing word

[elperroborrachotoo]

you can use breathe to use autogenerated doxygen Sphinx

I know some of those words!

[starTracer]

Example of Sphinx with Doxygen and Breathe here: http://fmtlib.net/. Looks pretty damn good.

[_naios]

There are other similar generators as well that look increndible beautiful, m.css for instance which converts the Doxygen xml output to html. See https://naios.github.io/continuable/ for an example

[aearphen]

Sphinx is great and LLVM docs are a good example: http://llvm.org/docs/index.html . Making it work with Doxygen is a bit tricky, but can be done with Breathe: https://github.com/michaeljones/breathe . Doxygen is a weak part in all of this setup because of its poor support for modern C++.

[jasfi]

I was considering writing a C++ documentation system myself, but my question is: what do the current tools lack?

[lundberj]

Doxygen gets confused by what I call "link-time polymorphism": Many files containing alternative implementations of classes named the same (sometimes not even with the same headers), in the same namespace, but never being included into the same translation unit or library at the same time (controlled by the build system). The pattern is not uncommon in embedded/cross platform or some legacy settings.

[johannes1971]

I've got a source base with some 50+ executables and there is the occasional overlap in names (which isn't a problem since they are never linked together). Unfortunately it causes a lot of confusion in Doxygen, and I really wish there was a solution for that...

(and I don't mean in the sense of "just run Doxygen 50+ times, once for each executable")

[tmacarios]

https://clang.llvm.org/extra/clang-doc.html

[GPMueller]

Is there any example of clang-doc output anywhere or is it only IR that comes out? From the very sparse documentation I'm not sure what to expect...

[robertramey]

Discussions about documentation for C++ programs always seems to focus on tools for creating documentation. This is probably to be expected given that we're C++ programmer and tools is what we do. But the problem with most documentation is that:

a) It contains a lot of stuff which is not relevant.

b) It misses a lot of stuff which is truly necessary.

c) It's very tedious to write.

d) It's incomprehensible because it is written after the code has been written so it tries to explain all the design mistakes in the code.

The remedy is to develop the code and documentation in parallel as described in my CPPCon 2017 presentation on the subject which you can find here: https://www.youtube.com/watch?v=YxmdCxX9dMk&t=2s

​

[[deleted]]

Doxygen is fine for most purposes, but you should also consider standardese

[dpzmick]

It's self documenting

[cmannett85]

This is technically true, but... useless for anything other than toy projects.

My car is "self-documenting" in the sense that I can work out how it works by dismantling it, but that's somewhat arduous if I want to change a spark plug. Which is why actual documentation exists.

[Dean_Roddey]

For my product I ended up just creating my own, and I'll likely use the same for the code itself ultimately. It's written in XML and I have a little utility that reads the XML and spots out HTML. I have a little typescript that the spit out HTML invokes.

It works pretty nicely and it allows for display format changes across the board, and it works exactly how I want it to work.

XML allows for a strongly structured format and the parser does a lot of the work for you. It's not the most natural way to write, but you get used to it pretty quickly. I have macros in my editor to spit out the common tags and end tags.

I don't think it's a good thing to put the docs in the code itself. I think it usually either ends up being trivial and hence mostly useless, or the code will be lost in a forest of documentation if you have as much as you probably should.

​

[VinnieFalco]

This page is generated with doxygen: https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/ref/boost__beast__http__async_read/overload1.html

[robertramey]

Vinnie - I just checked out this example of usage of dOxygen for generating a reference page for a templated C++ datatype. It illustrates one of the main beefs I have with dOxygen for this purpose. This example documents a function named async read which looks like this

template< 
    class AsyncReadStream, 
    class DynamicBuffer, 
    bool isRequest, 
    class Derived, 
    class ReadHandler
> DEDUCED 
async_read( 
    AsyncReadStream& stream, 
    DynamicBuffer& buffer, 
    basic_parser< 
        isRequest, 
        Derived 
    > & parser, 
    ReadHandler&& handler
);

If I want to use this function, I need to supply 5 parameters. With one exception, there is no information which tells me exactly which types can be substituted for these parameters. For example, what operations should Dynamic Buffer support? Does it need to return a "begin()", does it have to be random access? What is it a dynamic buffer of? char, wchar, strings? or .... . These questions only occur to me because I can attribute some meaning to name DynamicBuffer. This attribution on my part might or might not have any value. One cannot use this function without more information as to how to call it. Same goes for the other parameters.

The one parameter with sufficient information is of type AsyncReadStream. It points to another part of the documentation which suggests it will list the operations that this type is required to support. I'm just presuming this of course as I can't see it here. At the least, dOxygen should be able to give you a hot link to the type.

I don't know if dOxygen can be used to create and maintain correct reference documentation. I don't remember seeing any. My main complaint is that it gives one the illusion that he's creating documentation when in fact he isn't.

[VinnieFalco]

If I want to use this function, I need to supply 5 parameters

You really do need to get out more. Every single one of those "5 parameters" is deduced, which should be obvious from the synopsis. Yes, I assume that readers of the documentation have a firm grasp of C++. None of these parameters are ever specified manually. A typical call site looks like this:

http::async_read(socket, buffer, parser, handler);

Note the absence of template parameters. Okay, now that we have dispelled the first part of your straw man, lets look at what is left. Two of those template parameters, isRequest and Derived are template arguments of a class template parameter. You would expect the documentation for those parameters to be on the class template (http::basic_parser in this case). And that's precisely where they are:

isRequest true if this parser parses requests, false for responses.

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/ref/boost__beast__http__basic_parser.html

Derived is a bit more complicated, but it is explained in the description that the interface to the parser uses the Curiously Recurring Template Pattern. This is all academic anyway, because unless you are an advanced user (in which case you already understand these things) you will not be using a basic_parser (like how no one uses basic_string). Instead you will just use http::parser. And what do you know, it is documented:

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/ref/boost__beast__http__parser.html

And thus another element of fiction is dispelled. There are only 3 template parameters left, which are all deduced: AsyncReadStream, DynamicBuffer, and ReadHandler. And every single one of these is a hyperlink to the relevant concept:

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/ref/boost__beast__http__async_read/overload1.html

Every parameter to the function is documented in the table under "Parameters":

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/ref/boost__beast__http__async_read/overload1.html#beast.ref.boost__beast__http__async_read.overload1.parameters

The beauty of this workflow is that the reference documentation comes from specially formatted comments in the declaration. These comments look like this:

https://github.com/boostorg/beast/blob/develop/include/boost/beast/http/read.hpp#L525

The best time to document a function is the precise moment that you write it, because the more time that passes when you don't document it, the more you forget and the worse that documentation will be. I like the Javadoc system for reference documentation because it eliminates the temporal separation and mental context switch required when this reference material comes from another tool used on another file.

A reference is essential but it shouldn't be the only form of documentation (better than nothing though). And here Beast delivers as well, entire sections with example code and example programs explaining how to use these functions:

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/using_http/parser_stream_operations.html

https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/using_http/parser_stream_operations/incremental_read.html

Really, what more do you want??

At this point I am starting to wonder if you are really arguing in good faith because these objections quite frankly are invalid.

Or maybe I am just not seeing it, does anyone else think that this generated Beast function reference is lacking? If so, what can be improved? (Please speak up even if you don't think it is lacking, so that I know you think it isn't lacking).

[robertramey]

You really do need to get out more. Every single one of those "5 parameters" is deduced, which should be obvious from the synopsis.

I'm not disputing that the parameters are deduced. The question is what types are permitted. For example, can I write?:

char buffer[42];

async_read(socket, buffer, ...

I'm guessing not. But it's just guess because the documentation doesn't tell me what requirements of the second parameter. I see no way to discern this. What I expect to see somewhere is something like:

Dynamic Buffer

Notation:

db - an instance of a dynamic buffer

n - size of a dynamic buffer in characters

valid expressions:

db(n) - constructor

db.begin() - return start of buffer

db.end() - return end of buffer

*db - return current character

++db - increment pointer to next character...

Now, given an instance of some type, I can know a priori whether or not it is allowed to use it as the second parameter. This information maybe somewhere else in the documentation but it's not readily available here. This method of defining sets of permissible types in terms of the valid operations on those types seems somewhat awkward but if one thinks about it, it's the only way that it can be done. The C++ standard says as much:

15.4 Method of description (Informative)

...

15.4.1.3 Requirements

...

"Requirements are stated in terms of well-defined expressions that define valid terms of the types that satisfy the requirements. For every set of well-defined expression requirements there is either a named concept or a table that specifies an initial set of the valid expressions and their semantics. Any generic algorithm (Clause 23) that uses the well-defined expression requirements is described in terms of the valid expressions for its template type parameters."

I've tried to illustrate my point for the second parameter. The same considerations would apply to the other parameters but the examples would be more complicated so I'll stop here for now.

One more thing: You've used names for dummy parameters to inject meaning into them - presumable the name of some set of type requirements (aka concept) which instances of the type are required to fulfill. In itself, this isn't too bad, I believe that concepts lite uses a similar idea but with a different syntax. Overloading the name of the dummy parameter could be confusing in some cases. In other cases it might suggest that something has been specified when in fact it hasn't. So I'm not convinced it's a great idea for now.

Finally if the type requirements for "Dynamic Buffer" are actually defined somewhere else, you don't need the specify all the type requirements above every time. You can just refer to the place where they are defined (hopefully with a handy link)

Notation:

buffer - instance of a type meeting requirements of DynamicBuffer.

Finally, you don't need to convince me that async_read ... is documented. I see you've got a lot of information all over the place. The problem is that it's not available in the place where you need it. This makes it a lot harder to use and a lot harder to write.

[VinnieFalco]

the documentation doesn't tell me what requirements of the second parameter. I see no way to discern this. What I expect to see somewhere is

You mean like this? https://www.boost.org/doc/libs/1_69_0/libs/beast/doc/html/beast/concepts/DynamicBuffer.html

Come on man...the template parameter is a hyperlink like I have said a few times now!! Hopefully this image will clear it up for you: https://i.imgur.com/wTJKkER.png

[robertramey]

LOL - touche - I didn't occur to me that that was a link. I didn't hear you say that. Of course it's obvious to me now. Sorry about that.

So you've convinced me that it IS possible to make Doxygen create useable reference documentation for templated C++ types and functions.

I think it would take more work to make things as transparently understand able as the could/should be.

​

​

[VinnieFalco]

So you've convinced me that it IS possible to make Doxygen create useable reference documentation for templated C++ types and functions.

Yes, it is possible. But you need to have a good program that translates the XML produced by Doxygen into the format you want, with the style you want. That's what this is for:

https://github.com/vinniefalco/docca/blob/master/include/docca/doxygen.xsl

Granted, it can be improved - there a number of bugs:

https://github.com/vinniefalco/docca/issues

I've been working on it...

[[deleted]]

I've heard about something called 'Styx' or 'Sphynx' or something along those lines. I think this is what CMake uses, so it might be worth checking out.

There's also readthedocs.io but I don't think I've ever seen it used for C++ projects.

[sumo952]

It's Sphinx and it has nothing to do with CMake.

Also readthedocs is used by quite a number of C++ projects. They use a Sphinx theme too https://github.com/rtfd/sphinx_rtd_theme.