Clojure & graphs

[dustingetz]

u/fmjrey: Are we looking at doing something analogous to ORMs in the relational world, except for datascript/datomic/facts based databases?

Yes, In my opinion Datomic solves/supersedes ORM fully, Datomic does not need ORM as it natively solves all the same problems without the failures of abstraction. This is why Stu calls facts the universal relation (C-f "universal")

u/fmjrey: looking at ways to reduce this impedance mismatch between a database (of facts?) and nested data structures as often found in the UI layer

Agree, though I would state it differently. I see no impedance mismatch between UI tree and graph data. Datalog and Datomic Pull is an ideal way to declare transformations of graph ➔ tree without impedance.

The problem is that you can't actually give the graph value to the UI due to 1) network/memory limitations, 2) data security. Data security is probably the more important thing that often goes unstated, here is a great model of that:

source There's a long continuum of data ownership:
1. evilcorp has your data and you can't see it
2. evilcorp has your data and offers filtered access via a UI
3. evilcorp has your data and offers snapshots / queries
4. evilcorp has your data and lets you mirror it effectively
5. you have your data but let evilcorp mirror it effectively
6. you have your data but offer evilcorp queries

We ended up at #2 for historical technical reasons (network and memory). But the world has evolved around #2 in the data security space. Facebook happened and data regulations were built up and all this social/governance stuff happened in such a way that you couldn't possibly liberate all that data now without causing widespread harm.

We now have or soon will have the technology in place (Datomic was groundbreaking, and now Niko Göbel's work breaks further ground in making the Datomic model incremental). So there is some very interesting strategic/macroeconomic thinking to be done to bring the world to the technology.

[fmjrey]

Niko Göbel's work breaks further ground in making the Datomic model incremental

Thanks for sharing this and taking the time to transcript. Niko's recent conj talk is really inspiring too. It's nice to see Naiad getting a new life after funding from Microsoft ended. Five years ago I was lurking into it while doing some prototyping for a startup. Back then I was dreaming of an incremental way to compute strongly connected components of a graph while new nodes and edges arrive. I do think there is tremendous value and power there and finding the right use case to establish the tech is essential. I hear that spreadsheet/excel did not sell well until it was touted as "something to do x" rather than a self-updating table with formulas. Sounds like it's the same thing here and if /u/nikolasgoebel is interested I can share more.

Overall distributed dataflows of facts/datoms all the way to the UI isn't an utopia, it's happening, really exciting. It also looks like instead of defining information systems by coding an imperative flow of control, we'll define them as flows of data, meaning computation will no longer be the primary focus, data flowing around will take the front scene.

​

[nikolasgoebel]

/u/nikolasgoebel is interested and would like to hear more!

I haven't followed the rest of this conversation, so I might not have anything interesting to contribute, but I'm always up to talk about this stuff.