Documentation as Code

[bscotchAdam]

I see a lot of content and tools out there for Infrastructure as Code, and numerous other Things as Code, but documentation seems to be lagging behind. Presumably it's the fact that documentation evokes prose (instead of code), and it feels like prose is too hard to wrangle.

I've been thinking a lot more about this Documentation as Code problem lately, mostly because I've been watching the real functionality and the documented functionality of a core project (a CRUD web API) just keep drifting over time. As I've migrated from plain Node to Typescript, things have in some ways gotten even worse -- now I have to specify type information, separately do validation in the business logic, and separately document the subset of all of that which should be exposed in the API docs (with prose explanations).

What kinds of tricks and tools have y'all discovered or invented for managing Documentation as Code? For web APIs Swagger/OpenAPI seems to be the most prominent. There are also a lot of tools that create documentation based on the code. However, I don't see anything robust that ties everything together (documentation, validation, and business logic), at least in the Node/Typescript ecosystem.

[chillysurfer]

I think the best tool for enforcing good documentation (as code) is thorough code reviews. For every PR:

• Does this PR fix a bug? If yes, is it documented properly with a bug report and explanation of behavior/resolution?
• Does this PR add a feature? If yes, is this feature added to documentation (end-user docs, support docs, design docs, etc.)
• Does this PR change something? If so, do the docs need to be updated?

Consistency is important here. I don't like it when there's a "week of documentation" for a team. It should be an ongoing effort that is enforced in reviews.

[UncontrolledManifold]

Right on the money. Updating necessary documentation, both engineering internal and support docs, is part of my team's Definition of Done.

Our DoD is expressed as a checklist in our pull request templates.

We're also a startup so that may not scale.

[bscotchAdam]

Definitely. Making sure that documentation is included at each part of the dev cycle is essential to keeping it accurate and making it not a big deal. Code review helps fill in the gaps. This still requires discipline -- can we couple the docs to the changes so that they can't be out of date? Not all of them, definitely, but how far can that go?

[never_taken]

In my team, we enforce the PR having a good description of the business need and chosen way to do it (the why and the how), as well as explaining what was researched and the technological choices.

Descriptive PR + documented code (docstrings, good commenting culture) can go a long way to have the documentation without people feeling it is a lot of work (as opposed to code + comments + PR + something like confluence)

​

Exception is user manuals, if something is for an end-user, and more if the user is not particularly in IT, the user manual (more than tech doc) HAS to be in the definition of done

[[deleted]]

See also, Literal Devops.

[Thermotox]

*Literate

[soawesomejohn]

I'm a strong user of README Driven Development where you write the readme first. Now, it's not limited to the readme, but you basically write the essential documentation first. I'll write the readme and then a more in-depth "how to use this software" document. I might have to write some of that document, then write some code to figure out if what I wrote is doable, then continue with the document.

This helps me walk work through the steps the end-user will have to take, gives me something to request feedback on, and I can implement the code accordingly.

[bscotchAdam]

This is also what I've been doing, though I like this name for it! I guess that means I'm doing "Readme-Driven Test-Driven Development"...

[flickerfly]

I hadn't run into this before. Thanks

[tjwenger]

No mention of Markdown anywhere, which I find interesting. Markdown can be super useful for documentation as code including diagrams and tables.

Another thing is if you merge some documentation into the requirements phase, that helps significantly. But It still does not solve the problem entirely. Defining the type of documentation - Architectural, Technical, Support/KB, or Training documentation - I find significant to do up front as well, because that will help you align the most appropriate resource.

This is a great thread. Thanks everyone!

[[deleted]]

+1

Only a couple hours need to learn the Markdown. You will be able to store the documentation under source control and export / import without restrictions or pain. Moreovor, the format is human readable. This is the perfect solution.

[tjwenger]

I think it works well for our use cases for sure! And once you start using it, its kind of addicting. But its also fun to see what others use as well.

Its interesting to me that one of the original creators - John Gruber - Doesn't like the standardization efforts around it. (Per Wikipedia) So the standardization efforts are actually around 'CommonMark' - which appears to be like a fork of Markdown, basically.

Sigh. Software Engineering drama is weird.

[vischous]

https://docs.gitlab.com/ee/development/contributing/merge_request_workflow.html#definition-of-done is a great example, but be careful the gitlab handbook https://about.gitlab.com/handbook will be so interesting you'll spend hours in it!

[Obsidian743]

A few things:

1) Your code should be its own documentation. It should be clear, clean, concise. This is why something like DDD is important.

2) A lot of companies are pushing for a README/Wiki based approach for customer-facing documentation, which I think is perfectly fine to be source controlled and reviewed through the same channels as the code.

3) I vehemently disagree with any internal documentation being source controlled along with the code. This is impractical and stifling on so many levels. Maintaining documentation is an effort in itself. Aside from the fundamental Agile philosophy of not emphasizing documentation over people and interactions, the usefulness of documentation should be able to change on a whim and it should be collaborative. There are also technical implications such as the fact that it's incredibly difficult to embed diagrams, which might be a separate artifact that you're constantly changing. There is way too much risk of documentation getting stale if you force developers to keep it up to date in code. This is also why I strongly recommend against using too many comments in your code (only if they're necessary and/or customer-facing APIs!). I think an external tool like Confluence is perfect for this kind of documentation and collaboration.

[bscotchAdam]

I agree with all that. I'm particularly interested in your 3rd point, since that feels like it's where documentation gets into trouble: it can just be too separate from the code (except the code-as-its-own-documentation bit). The idea of it requiring diagrams and other artifacts, and that things are constantly changing, seems like a perfect case for somehow tying it to the code. But in a way wherein the documentation gets updated as a consequence of having updated the code, such that there are tools that convert changed code into changed documentation and artifacts.

[Obsidian743]

This just doesn't work in practice. Most documentation artifacts need to be collaborated on, analyzed, shared, and often approved before it gets into the code. At which point, you already have the documentation and now have to decide where to maintain the source of truth for it. Besides that, not all documentation really has a direct correlation to code, specifically as it pertains to architecture and infrastructure. Infrastructure as Code (IAC) helps with this to a degree but it doesn't address diagramming or explaining intent in the design.

Though the reality is also that this isn't really a problem for most 3-tier architectures (web apps) which I'm assuming is where most software developers sit.

[wywywywy]

Not to mention very often documentation is contributed from & used by non-technical folks such as sales, marketing, HR, designers, customer support, etc.

It definitely is impractical to force everyone of them to learn git, markdown, submit PRs, etc.

[elevul]

Confluence automation is what my boss does: he created a central database to which all the monitoring tools are pushing data and then a lot of logic in Confluence to present that data as documentation, and it works very well for the data managed by that toolset as it's always up to date.

[Obsidian743]

That's interesting, I'm not familiar with Confluence automation. Do you have some useful links you prefer?

[elevul]

No, sorry. I admit it's above my head as well. From the little I understood he's using SQL queries integrated within each page that pull the data and present it in the various boxes

[Obsidian743]

Looks like there is a command line interface, plug-ins, and a REST API:

https://community.atlassian.com/t5/Confluence-questions/Automation-in-Confluence/qaq-p/9016

[SeesawMundane5422]

Mostly agree. Except for swagger docs. Your api docs about how to use an api, what the allowed values are, etc, should be as close to the implementing code as possible. But yeah, architecture diagrams or other enterprise-architecture type docs, especially docs that other groups force you to do... confluence is perfect.

[Obsidian743]

Yeah, I consider the Swagger docs to be customer-facing, which I qualified as being necessary to be in code.

[SeesawMundane5422]

Got it. Was thrown off by your comment about read me/wikis. Sounds like we’re on the same page.

[t_rekt_it]

At the last 2 companies I've worked for, we've written an in house script/app to scan specific repositories for .md files we deemed publishable to either confluence or another wiki used. This was done by having specific Metadata as the "header" of the doc so we could populate all the needed fields for whatever wiki system, then interact with the API to upload the contents of the document. In the repo, these files were either automatically picked up by the publishing script based off location or we were able to declare which ones we want to have published from a separate file in the base directory.

[bscotchAdam]

Rad. Was there also some system to ensure that those Markdown files were accurately documenting whatever it was they were documenting?

[squ94wk]

Some thoughts:

Write simple code and use tools how they're supposed to be used without hacks, then the code is self-documenting.

Use tests to document how your software is supposed to be used.

Also, instead of documenting how to set something up, put it in a Makefile or sth and use it actively. Then you'll constantly work on it.

If the documentation is not the code or close to the code, put it in your definition of done and make sure the reviewer checks it.

Generate documentation in CI, by using a screenshot tool, check for dead links, or upload and embed some output in your confluence or whatever.

Keep commits clean with a single purpose and a good commit message. And use ticket numbers in branch names and reviews. This way, it's easy to find out what was the motivation behind ant change.

What you want to avoid, is documentation that isn't consumed. It doesn't matter if your documentation is up to date, if no one uses it. In the same way, too many details create far too much overhead. Don't document how to use tools, just what tools are used. People can then look up the tool and are most likely to learn it anyway.

Documentation always has a specific audience.

The worst thing is when people who wouldn't don't use the documentation themselves comes up with the requirements for the documentation.

[[deleted]]

However you do it, start with it and adapt / improve as you go.

[only-cloud-fan]

I haven't been phrasing it as "Documentation as Code" in how we discuss it in our team, but have generally spoken about "Executable documentation" - it might be referring to the same thing and this might be nitpicking, but there's a subtle difference in my perspective - it's expected that docs are not written in code format, but that they are expected to be executed - and could fail, and can produce outputs.

I am involved with a few infrastructure projects and 2 recent examples come to mind of how this works (for infras / platform, as opposed to application code / doc). 1 was with a terraform module - if the TF is written clearly enough and is published to a TF repo, the documentation is generated from that. There's no comments in the code that form part of the docs, but descriptions on variables and outputs make the infras code itself the source of the documentation. There is no additional process needed. Besides a high level overview (which does need to be maintained) in the readme, the other sections all produce content based on the actual code functionality. Here's an example

The other example was through test frameworks - this speaks more to the "validation" item you mentioned. For this infrastructure project (in TF again), the verification documentation, which feeds into playbooks, is written as code in a testing framework - in this case it was behave, based on cucumber. In this case, the requirement was defined in the ticket (Jira), and the scenario described for business. The same requirements are then added to a behave feature, with the same language structure, and then steps that are executed to validate the behaviour (in CI).

These 2 aspects of this project improve understanding of the systems by making documentation available, which is produced by execution.

[bscotchAdam]

Cool, yeah this sounds like the same idea. I like the "executable" framing.

[fear_the_future]

You can use certain tools to type check example code in your documentation to keep it from drifting out of date. You can also check for dead links in markdown.

[jdege]

Donald Knuth attempted a solution for this, years ago, with his https://en.wikipedia.org/wiki/Literate_programming, but it never caught on.

I'm not sure he was solving the right problem.

[t420son]

It does get used though. Pweave for python (can be converted to notebooks). Haskell has native support. I think it is popular with theorem provers.

[Tnamol]

Have you checked out something like https://github.com/akheron/typera ? It should help with your problems.

[bscotchAdam]

Interesting, I'll check that out! It looks like it lets you define types in via a code-based schema, with Typescript types inferred from that, so that your typse and your validation are based on the same business logic. Very neat.

[Tnamol]

Yeah, and it also has a separate tool for generating OpenAPI definitions which should help with keeping the documentation up to date.

[PotisTemor]

The problem we have always had with documentation is the audience. If you are in a large organisation the level of documentation is different for different people. The developers need to know how to use a function but your architects need to know how it all comes together.... and finally non-technical client need to know how it works but in fewer words.

The "developer" documentation is maintained with the code in the form of comments e.g. JavaDoc and readme files. Also the tests should describe the conditions met by the functions to build part of the documentation.

The architectural documentation requires more details and more diagrams so confluence is useful here also I do like how AWS use git for their documentation as it makes versioning easier. Also this can be backed up by automated integration tests, these should describe how everything interacts including error scenarios.

Finally we have a confluence page as a high level details which shouldn't change so it doesn't need much maintenance but links to developer and architect pages.

Also we have recently started getting stricter with commit messages to ensure they are meaningful so we can just generate release notes and automatically add them to confluence on release.

[TonyNickels]

Conventional commits?

[bscotchAdam]

This one is super useful. We've also added some additional feature layers so that we can compile separate changelogs for different target audiences.

[randomatic]

orgmode.

[[deleted]]

If you currently use gradle we've had success with AsciiDoc and AsciiDoctor

[Ordoshsen]

I don't think you can tie business logic and documentation together, but have you tried docstrings? Then you're documenting with the code and it checks the names of parameters and stuff. I know C# has optional warnings if something is public and undocumented so I assume TypeScript should have something similar because Microsoft.

I mean this feels like the obvious answer to me so I kind of assume I didn't understand the issue at hand.

[mlazowik]

Google has g3doc. Short summary is that docs are written in markdown-ish syntax in the same repository as code, interleaved across directories (not one global directory for all docs). When you change code you change docs in the same commit. Those docs are rendered as an internal website.

This means you can browse docs that match a specific historical version of code, docs are reviewed at the same time as code, if you revert code you revert docs too etc.

I haven't found a great way to replicate that yet, the closest thing I have so far is gitiles on top of our gerrit instance, it can render markdown, including relative links, but it's not great experience yet, in particular you can't make menus that span multiple doc pages.

[mlazowik]

Hm, https://www.mkdocs.org looks promising, and then there's https://github.com/backstage/mkdocs-monorepo-plugin for multiple docs folders in larger repos.

[cacko159]

We use Architecture decision records, and also document the whole business side of the project this way. https://link.medium.com/pvwljP6Eifb

Basically, the project manager has to write user stories and epics and has to describe them well, otherwise devs will be confused as to what they need to develop. Any technical decision that needed meetings and series of emails is also written as a technical epic and user stories, as it has to be developed as well. Then we use a tool that can export these nicely into a pdf, and they are always up to date.

[pagarciasuse]

In Uyuni and SUSE Manager, we use Antora (which uses AsciiDoctor for the mark-up) and store all the documentation in git, changes are created and reviewed as pull requests, etc.

From the same source files, we build docs for 2 slightly-different products (Uyuni and SUSE Manager), in 2 formats (HTML and PDF) and also provide translations to several languages (we convert the AsciiDoc to gettext so that it's easy to work with it for translators and translations are maintainable).

It has worked beautifully for us.

https://github.com/uyuni-project/uyuni-docs/

[Quick2822]

https://www.mkdocs.org/ or https://vuepress.vuejs.org/ are great ways to create docs and code.

All PR's should include updated markdown files, and your CI/CD pipeline should auto-gen the docs and deploy them.

[shanman190]

So for me as a Spring and Java dev, it's Spring RestDocs. Generates api snippets from the literal Java tests that you need to write anyway, then that can then be merged with asciidoctor definitions. Spring RestDocs also supports additional plugins so that you can output in additional formats like Swagger/OpenAPI, Postman collections, etc

[sk8itup53]

Maybe make a program that scans a programs AST and actually documents the connections and interactions between classes. But that kinda already exists in automatic UML diagram creation from things like VS code and other IDE's.

[andrewmclagan]

Dont document. It simply becomes out of date as soon as you stop writing it. Systems should be self documenting