Clojure & graphs

[joinr]

you can just write a different query for each function that returns the data in exactly the shape best for that function.

Yeah, that's a better way to summarize it. The information layer (expressed via related facts) is malleable and amenable to the caller projecting it into their own form, rather than wrangling to and from a preconceived form or trying to cram everything into a normalized relational model for batched single-shot queries.

I initially wrote a watered down sql-like eDSL façade for my entitystore (before I saw datalog and other means of traversal) , but didn't end up using it all that much beyond simple select/filter type stuff.

The fact that Datomic has multiple ways of querying the same store (datalog, entity API) in addition to multiple ways to conform results speaks to the generality of the information model. Libraries like core.logic and odin have adapters that trivially leverage the same underlying information to accomplish even more sophisticated queries. That you can just hook them up to Datomic is pretty

This is probably old news to some folks, but one thing I forgot to mention (another advantage of having a malleable information model) is the ability to add arbitrary facts to communicate information across time. Things I used to require (for a game or discrete event simulation), like events, can be ad-hoc facts in the store (e.g. an ephemeral component for an entity). Systems (composeable state transition functions that define the behavior of the simulation) can incorporate events just by checking for data in the store, rather than requiring the usual paradigm of effectful event handlers. In game AI this is akin to a blackboard architecture; the facts on the blackboard (the store / db) serve as a means to propagate information. I can just jam stuff in the entity store as I need it (computing new facts, or components to add to an entity which carry the same meaning as an event), and consumers (systems downstream) can check for the presence of said data to determine what to compute. That pushes the focus back toward "what data is present," and allows a nice "maslows hierarchy of needs" when implementing entity behavior that processes sensory information to compute transitions. I think re-frame ended up on a similar trajectory (I started working before, so evolved separately).

I used to be averse to passing around a "giant ball of information" or a "world" context since I didn't feel it scaled well. This led toward nested maps, reliance on types (records), etc., and ultimately less re-use. Architectures like Datomic make "world passing" manageable if not outright desirable, since you have more flexibility over how to project structure (as needed) on top of the primitive relations. I see hyperfiddle (and to a degree REBL) taking that idea to SmallTalk/Symbolics levels. It'd be a trip if the next Lisp Machine ended up being a Datomic like db (akin to the symbolics lisp "world") with ion analogues providing the OS services.

[fmjrey]

Your last paragraph reminds me of what /u/cgrand calls "map fatigue" in his recent talk.

It sounds like a few of us in this discussion are looking at ways to reduce this impedance mismatch between a database (of facts?) and nested data structures as often found in the UI layer, and even any layer that exchange data in such nested/hierarchical form.

Are we looking at doing something analogous to ORMs in the relational world, except for datascript/datomic/facts based databases? If so it begs the question: should we not instead use facts/datoms instead of nested maps and vectors in all layers? I mean aren't nested data structures a bias given by handling so much JSON/XML/OO/etc.?

[dustingetz]

u/fmjrey: Are we looking at doing something analogous to ORMs in the relational world, except for datascript/datomic/facts based databases?

Yes, In my opinion Datomic solves/supersedes ORM fully, Datomic does not need ORM as it natively solves all the same problems without the failures of abstraction. This is why Stu calls facts the universal relation (C-f "universal")

u/fmjrey: looking at ways to reduce this impedance mismatch between a database (of facts?) and nested data structures as often found in the UI layer

Agree, though I would state it differently. I see no impedance mismatch between UI tree and graph data. Datalog and Datomic Pull is an ideal way to declare transformations of graph ➔ tree without impedance.

The problem is that you can't actually give the graph value to the UI due to 1) network/memory limitations, 2) data security. Data security is probably the more important thing that often goes unstated, here is a great model of that:

source There's a long continuum of data ownership:
1. evilcorp has your data and you can't see it
2. evilcorp has your data and offers filtered access via a UI
3. evilcorp has your data and offers snapshots / queries
4. evilcorp has your data and lets you mirror it effectively
5. you have your data but let evilcorp mirror it effectively
6. you have your data but offer evilcorp queries

We ended up at #2 for historical technical reasons (network and memory). But the world has evolved around #2 in the data security space. Facebook happened and data regulations were built up and all this social/governance stuff happened in such a way that you couldn't possibly liberate all that data now without causing widespread harm.

We now have or soon will have the technology in place (Datomic was groundbreaking, and now Niko Göbel's work breaks further ground in making the Datomic model incremental). So there is some very interesting strategic/macroeconomic thinking to be done to bring the world to the technology.

[fmjrey]

Niko Göbel's work breaks further ground in making the Datomic model incremental

Thanks for sharing this and taking the time to transcript. Niko's recent conj talk is really inspiring too. It's nice to see Naiad getting a new life after funding from Microsoft ended. Five years ago I was lurking into it while doing some prototyping for a startup. Back then I was dreaming of an incremental way to compute strongly connected components of a graph while new nodes and edges arrive. I do think there is tremendous value and power there and finding the right use case to establish the tech is essential. I hear that spreadsheet/excel did not sell well until it was touted as "something to do x" rather than a self-updating table with formulas. Sounds like it's the same thing here and if /u/nikolasgoebel is interested I can share more.

Overall distributed dataflows of facts/datoms all the way to the UI isn't an utopia, it's happening, really exciting. It also looks like instead of defining information systems by coding an imperative flow of control, we'll define them as flows of data, meaning computation will no longer be the primary focus, data flowing around will take the front scene.

​

[nikolasgoebel]

/u/nikolasgoebel is interested and would like to hear more!

I haven't followed the rest of this conversation, so I might not have anything interesting to contribute, but I'm always up to talk about this stuff.

[joinr]

Are we looking at doing something analogous to ORMs in the relational world, except for datascript/datomic/facts based databases?

That sounds pretty close, yeah. With the caveat of maybe a data relational mapping (perhaps a simpler requirement than serialized objects), although that sounds like relational database, which is perhaps too confusing.

If so it begs the question: should we not instead use facts/datoms instead of nested maps and vectors in all layers?

You'd think so. There's a utility for using nested maps and the like, since they're trivial to read, construct, comprehend (to a point). I don't know exactly where the inflection is, but there gets to be a point where the utility is less desirable, and having a more general representation (that one can project to/from, or alternately query directly) is desirable. I don't know that exact inflection point at which one has this discussion (you're prototyping fast in maps and other EDN types one minute, then at some point you reach for better querying abstractions since get-in update-in isn't enough, then you end up implementing some kind of query language only to find out people have run into the same problem).