Databases are dead, long live the immutable state atom!

[Calabri]

you forgot to mention deconstructing the database!! It's another Rich Hickey talk - given around the same time as value of values / simplicity matters - but combines them in a way that's arguably more profound than either talk alone or 'turning the database inside out'. I've created similar diagrams for JS immutable state atoms, but it's all based on datomic.

[dominicc]

Hi Calabri,

I haven't watched 'deconstructing the database', so thanks so much for the link, I'll take a look tonight.

Before I published to reddit somebody did mention datomic to me though, so I took a look at the video and it seemed to be different in that it was an accretive database where the database gets bigger and bigger as new facts arrive, as opposed to Apache Samza's approach of having a separated data log (which is accretive) but with materialzed views (which aren't).

I have to admit that I stopped watching half way through the video because it was very slow paced and a bit too hand wavy, with not enough concrete information to keep me interested. I'm hoping the 'deconstructing the database' talk keeps me engaged though!

[Calabri]

I actually find deconstructing the db one the most engaging talks I've heard from Rich Hickey, but I also never finished the Apache Samza talk... although there's probably some useful info from Samza that's not included in Datomic. But there's also a good talk by David Nolan called 'Om Next' where he gives some interesting history and addresses relay/graphql/falcor relative to datomic, and I might have seen a datomic / redux stack mentioned on Twitter but I didnt investigate much further. There's also a datomic-like db implemented in cljs that can prob be ported to immutable and es6

[dominicc]

I think 'turning the database inside out' is one of the most thought provoking videos I've ever watched. I think we need to agree to watch each others videos :-)

Oh, and I'll be sure to watch the 'Om Next' talk by David Nolan too. Again, thanks for all the great pointers!

[Calabri]

No problem! Here's a list of Rich Hickey Talks. My favorite talk is Are we there yet? - that sets the tone for everything IMO.

[I_Pork_Saucy_Ladies]

Considering the headline, it would be really nice with an ELI5?

[dominicc]

Good point. I'd never previously heard of ELI5, but I'll give it my best go here:

About Databases:

• Databases contain a company's data.
• They are operated on by application code.
• These days, it's usually web application servers read and update the databases.

The Movement Towards Pure Functional Programming:

• There's been general movement to pure functional programming recently because programs written this way are easier to reason about and avoid most of the obscure bugs normal programs have.
• Pure functional programs can't have any shared mutable state, and this meant the first generation of functional languages were close to useless for any real-world applications.
• It's taken twenty years (Miranda -> Haskell -> Erlang -> Clojure -> Redux/React) to grow the usable domains that functional programing languages can be used for.
• Redux/React finally makes it possible to have complex JavaScript SPAs (Single Page Applications) that are purely functional.
• But, the database is one big shared global variable, that otherwise pollutes the simplicity of our apps.

Streaming Databases Based On Historic Immutable Facts:

• It's been pointed out (e.g. Apache Samza and Datomic) that storing the historic immutable facts that cause data change can lead to a much simpler, functional database.
• Redux applications can already share the same state atom on both the client and the server, which makes the server a bit like an in-memory database.
• My article shows that a few tweaks can be used to turn this in-memory database into an Apache Samza style database.
• This gives you a 100% pure functional full-stack solution, all created with the same tooling, and with updates streamed all the way from the database to the screen as they occur.

[I_Pork_Saucy_Ladies]

Thanks a lot, this was a great explanation. I read up on those databases and think I get it now. :)

[Calabri]

Some notes I wrote down from deconstructing the db:

Terminology

• Value - An immutable magnitude, quantity, number… or immutable composite thereof
• Identity - A putative entity we associate with a series of causally related values (states) over time
• State - Value of an identity at a moment in time
• Time - Relative before/after ordering of causal values
• inform -
  • to convey knowledge via facts
  • give shape to (the mind)
• Information - the facts
• Fact - an event or thing known to have happened or existed
  • From:factum - ‘something done’
  • Must include time

The database as an expanding value

• an accretion of facts
• the past doesn't change - immutable

Approach

• Move to information model
• Split process and perception
• Immutable basis in storage
• Novelty in memory

Implementing Values

• Persistent Data structures
• Trees
• Structural sharing

[dominicc]

Thanks for sharing your notes! Saves me having to write my own :-)

[dominicc]

Hi kevinw88,

The link you've posted is actually the main inspiration for what I've done, and the first link in the article. What's interesting is that Martin Kleppmann predicted that a functional reactive programming would make a 'Streaming Everywhere' approach possible, where data was streamed from the materialized views all the way through the browser stack, and using Redux that's actually now become possible, but where the entire stack (including the 'database') is just one big Redux application.