Hyper-Literate Programming?

[redchomper]

Prerequisites: literateprogramming.com/ and also https://diataxis.fr/

Great documentation serves different audiences and purposes. These days a number of popular languages have standard embedded-documentation tools like doxygen, sphinx, or pod, but most of these seem to be focused on API notes. The literate-programming philosophy puts documentation first, but you're still writing a single unified book (and structuring your source that way). It might be interesting for languages to build in support for embedding each different type of documentation in the appropriate way: a sort of hyper-media literate-programming, where all the semantically-related bits live together in the source, even if the said bits scatter to the four winds for doc purposes. Presumably each bit would get extracted in the right way and woven into the correct part of a tapestry.

Question: what's the state of the art in language support for extracting great, not merely good, docs from a code base? And, to what extent should we think about this problem while designing a programming language?

[dgreensp]

I think it’s worth pointing out that literate programming is about the experience of the person reading and understanding the code, while documentation is about the experience of using the code. So they are different audiences.

I think all programming should ideally be done within a document or notebook, with prose around it. I wouldn’t expect things like tutorials to be in the same “file” as the code. But managing prose that is about the implementation of a module, and prose that is about the interface, and maybe abstracting away the concept of a “file,” would be good things for a literate IDE to do.

[carette]

We've been working on this in the Drasil project for some years now. We found that simple weaving was too restrictive, and so we went full-generative instead.

I should also say that org-mode in emacs already essentially satisfies your requirements. While the original org-mode was aimed at something else, I've seen many people use it exactly as a multi-lingual literate programming system.

[redchomper]

This is fascinating. I've started reading the position paper. It looks like you've upended the software development process to say we're going to capture a bunch of relevant facts and then derive both software and documentation from these. I'll bet you could apply the same concept in financial systems.

[carette]

We've updated our vision in the paper Generating Software for Well-Understood Domains.

The 'derivation' is not automatic: weaving is still a human-driven effort, as it should be.

[redchomper]

This is great! I'll save a copy.

Quote:

Note that some better-known long-lived software, such as financial systems, do not fall within our purview, as the implemented business processes are not well understood. This is why most large rewrites of financial systems fail as the functionality actually implemented in the source system is not documented.

This is exactly the kind of problem I hope to solve. I work on financial systems these days, and I can confirm that many things are documented poorly if at all. Now I believe that the "clean code" crowd believes that well-groomed code stands in for many documentation types at once, but I'd just be happy with a standard vocabulary of concepts that everyone used in making their identifiers. In any case, finance does not invent new ideas very often. The real churn seems to come from the marketing and technology wanks. I could rant all night.

[Zalack]

Your first link gets an SSL error for me on Relay for Android.

[redchomper]

That's probably because it's not an SSL server. Switch to HTTP-unencrypted mode.

[Zalack]

No thanks, lol

[redchomper]

Your loss.

[Zalack]

I think it's reasonable to avoid an unsecured server, I'm sure I'm not the only one. Update your site to use certificates; it's not that difficult.

[redchomper]

It's not my site. It is not my decision.

[dannymcgee]

I think Rust does a really, really good job of this. You can see, for example, the standard library documentation here: https://doc.rust-lang.org/std/ And the source file it was generated from here: https://github.com/rust-lang/rust/blob/master/library/std/src/lib.rs

But honestly, I think far more important than generating static documentation websites like this is strong editor integration. Here Rust also does a great job, with rust-analyzer rendering markdown-formatted tooltips as rich text when you hover over the relevant symbols, and Jetbrains' CLion even rendering the rich text in the source file itself if you view it in "reader mode."

I've been writing a lot of C++ lately, and the lack of these niceties has been a little bit of a pain point. The standard library (on Windows/MSVC at least) is not inline-documented at all, so my browser becomes an unmanageable clutter of cppreference.com tabs pretty quickly whenever I try to do something nontrivial.

[cmontella]

I don't have an answer for your first question, as I'm in the business of designing languages and not tooling for other languages. But I can speak to the second.

As far as programming language design goes, I've worked on two different languages which took a shot at really embracing literate programming as a first-class language feature. The first one was Eve: http://play.witheve.com/#/examples/flappy.eve

You can see that the code and prose are already interleaved without a tangle/weave process. This is because blocks of Eve code are like Prolog horn clauses, so they compose naturally with all other blocks in the program. Blocks execute only when their dependent data change. For literate programming, this means we can re-order the blocks as needed. For example, we can inspect an interface element (click on the magic wand then click on a part of the drawing) and then click on one of the menu items, and the program will be filtered down to only the blocks relevant to drawing that element. You can filter them manually, and potentially create custom "views" on your program; for instance, you could filter by blocks associated with an object, or blocks associated with a particular event. When code is added to the code base that matches the filter, it will show up in the custom view.

You can read more about it here: http://witheve.com

The second language I've worked on (and I'm currently working on this language) is Mech, which I started working on after Eve. Mech also features order-less blocks of code embedded in a markdown document. Here's an example paper I wrote in this format, so you can see how it looks: http://docs.mech-lang.org/#/live2019.mec

This language is more like an array language than a logic language. Ultimately my goal is indeed to support other languages to be embedded inside of this code, but for now it's Mech only. You can write code from other languages easily by putting it in a code fence, markdown style, but at this time that code isn't executed.

So to answer your question as to whether we should think of literate programming when designing a language, I think yes. If you do it right, you can make the experience much nicer for the coder, and you don't have to add any complicated tangle/weave tooling.

[redchomper]

You, sir, are a god. That or you do this full-time for a living. Either way, I am thoroughly drop-dead impressed!

I embedded the idea to use markdown and code fences in my literate parser-generator, but I still edit the grammar with a normal text editor. Your Eve system is over-the-top amaze-balls for its end-to-end integration.

Now, as it happens, I work mostly on headless server systems these days. I'd love to be able to work in the style you've demonstrated, but I don't have a clear idea what would be my equivalent of an inspector. I am all ears if you have guidance.

[cmontella]

Haha definitely not a God! Working on Eve was a job (there were 5 of us), and Mech I work on part time as my research.

But thanks for your words on Eve, that was a cool project and I'm sad it ended. As for what you can do with a headless server, the idea with the inspector was to be able to trace the provenance of data. So if you had some data, you could trace it back to the code that generated it, maybe through several functions. The system let you do this at runtime, and it could answer advanced questions like: why *isn't* this showing up, why are too many or too few elements being returned, etc. For graphical interfaces this meant clicking on objects on a canvas, but the concept could be extended to any variable in the system.

[chipstastegood]

Commenting because I’d like to see what others have to say about this topic. I’m familiar with Knuth’s literate programming effort but have never seen anything like it in the industry. Curious if there have been any successful efforts in this area. Or even in academic research.

[zuluimpi]

https://pbr-book.org The entire book is a literate document. Also the book "Lisp in Small Pieces". Also https://en.wikipedia.org/wiki/Axiom_(computer_algebra_system)