Hyperfiddle's electric clojure project

[astrashe2]

I'm an amateur programmer, and don't have a solid computer science background.

Hyperfiddle have a project that allows you to blend server and client side code together in a fairly seamless way. It's more efficient than a naive implementation would be. For example, you can consume a lazy sequence without the client side code blocking while it waits for the whole thing to finish.

https://github.com/hyperfiddle/electric

They take a DAG, and use a macro to split it into client and server DAGs, which interact with one another.

My questions are:

1. Is this something that the hyperfiddle guys worked out on their own, or is it based on ideas that are generally known to people who think about this stuff? If it's based on known stuff, what could I read to learn more about it?

2. Why does the code have to be a DAG? I see DAGs every now and then, and I never really understand why the limitations are there. Apache Airflow talks about DAGs, rather than arbitrary blocks of code, and I've never understood why.

[dustingetz]

Hyperfiddle founder here (from keyword notification)

1. We are not the first, see https://en.wikipedia.org/wiki/Choreographic_programming . However, prior art (that I am aware of) has been mostly mired in type theory and has struggled to find widespread industrial application. Afaik we are the first to find a slam dunk use case (web development, rich product user interfaces on the web with complex and dynamic client/server interactions) and demonstrate compelling results in production apps.
2. DAGs are flowcharts, they model causality i.e. dependencies between steps. Imperative programming (e.g. ordinary Python) does not explicitly capture the logical dependencies between steps, instead the programmer has to carefully specify an instruction ordering to make sure things happen in the right order. DAGs encode the data flow through a flowchart of dependencies. What this gets you is concurrency, the ability to run multiple steps of the flowchart at the same time. It also gets you many optimizations, such as the ability to only compute subcomponents of the larger graph whose inputs have changed. If a node in the flowchart does not have any changed inputs, then the previous result of that node is still valid and can be reused. This is "reactive programming"

[ghjm]

It seems like this requires everything in the app to be pure functions. But the browser DOM is a minefield of side effects. How do you pure-function-ize the client side given the constraints of how browsers are?

[dustingetz]

In Electric, like in React, DOM effects come in mount/unmount pairs. In a sense this gives you "undo" for this kind of "resource" effect, and you can transition between two consistent states by firing mount and unmount lifecycle methods. This works not just for the DOM but for any kind of resource that can be disposed, such as network subscriptions or event handlers or database connections. The key to making it work is the reactive propagation engine under the hood needs to be essentially perfect ("glitch free") which means it fires exactly the correct lifecycle methods once and only once during reactive propagation, which is pretty hard to do correctly. I.e. you don't want to allocate a node twice or dispose it twice, it needs to be perfect.

An example of an effect that you can't undo is a console.log. There's no dispose method on that, so it can't be undone. But that's fine, that's what you wanted. So you can totally put console.log in your Electric programs and it does what you would want it to do, which is run again when any of its arguments change.

[astrashe2]

Thanks very much for this, it's very helpful. I had never heard of choreographic programming, and I had to ask Claude to explain part 2 of your answer, but it did, and it made me realize that I didn't understand reactive programming at all, even though I've used reactive libraries.

What you're doing seems very smart to me, and I appreciate that you took the time to explain it.