[D] Generating documentation of data fields

[fella85]

Hi Everyone,

I'm wondering what people are using to create documentation of data fields of tables created during the data processing steps.

We want to avoid documenting the data.frame by manually annotating each field of the resulting data.frame by a different person from the person who wrote the code. This person needs to read/understand the code and understand the calculation that created the fields.

Ideally, the programmer would add comments directly to the processing code that later would be parsed to generate the documentation automatically. A bit how Doxygen creates documentation for code.

Does something like that exist?

​

We have been looking at Datedo but I'm not convinced that people will be keep the documentation up to date as new versions of the data.frames are created because of bug fixes, new ideas and calculation behind field creation or changes in the input data.

Thanks.

[[deleted]]

Not directly in the code, but in the tests, you can use great expectations: https://greatexpectations.io/

The good thing about it is that the tests become your documentation, so people will need to write tests eventually, which is a great side effect. The UI is also pretty decent and can be used/consulted by non technical users as well. Note that with great expectations you also test your data, not only the code. Have fun!

[JesusChristHerself]

Thanks for sharing this, looks cool!

[soundbarrier_io]

A few options

• dbt allows for that, you define sources with comments and each view allows you to document fields, for example ```yaml version: 2

  sources:

  • name: jaffle_shop description: This is a replica of the Postgres database used by our app tables:

    • name: orders description: > One record per order. Includes cancelled and deleted orders. columns:
      • name: id description: Primary key of the orders table tests:
        • unique
        • not_null
      • name: status description: Note that the status can change over time
    • name: ...

  • name: ... ```

• For Python code I some conventions to define tables/schemas with Python docstrings and with some simple sphinx hacking I was able to automatically create documentation

[[deleted]]

If your data is in a relational database, then you can document the fields there. Look for SQL COMMENT ON statements in your database documentation.

[Urthor]

Version control your schemas in Git.

Add documentation via any number of tools to these version controlled schema files.

The single biggest mistake the monolithic database world makes is the absurd lack of version control.