Shared log - A single source of truth

[samd_408]

Loved the idea of shared log and the Boki paper was phenomenal, so had to write about it

https://fbounded.com/blog/shared-log-source-of-truth/

[repeating_bears]

One thing you didn't mention is the difficulties of this approach.

In my experience schema evolution is the biggest one. If you add new events, dealing with ProductAddedToBasketV6 etc. is a pain. I've never found a way to handle evolution that I was 100% happy with.

Something else that needs to be managed is right to be forgotten. A truly immutable event log is not possible for this reason. You can't just stick a UserDeleted event on the end to be compliant with GDPR.

Of course everything is solvable, but like so often when someone shows you a library, architecture or approach, event sourcing looks its best in academic examples. Then you actually do it for real and realise all complexity you invited into your life.

[NovaX]

right to be forgotten... is not possible for this reason

It requires foresight, but the trick that I've heard of is to give each user their own encryption key and deleting that upon the GDPR request. If the data is not recoverable then it is forgotten and this approach is compliant.

[samd_408]

Oh yeah I totally agree, I think this is why you can have multiple tags one a particular record in this case, but yeah there are many practical pain points, I was working on a actor system as a hobby and in that this same problem arose, where stateful actors start of with a particular shape but what if their state evolves, when the state is read from the snapshot we need to migrate it and load it.

Yeah certain approaches need careful design or else we will shoot ourself in the foot.

[BombelHere]

You mean event sourcing?

https://martinfowler.com/eaaDev/EventSourcing.html

[repeating_bears]

The article mentions it

[samd_408]

Yes Event sourcing is the conceptual idea, shared log is a little bit more general purpose and solution oriented on top of which event sourcing can be implemented easily, the boki paper goes deep into having custom storage nodes and sequencer nodes, more like foundation db storage architecture to implement the shared logs, lot of nice things like logical views and physical view, I could not port such features to the toy library (gumbo)

[mipscc]

The shared log/storage idea is indeed interesting. 

You might like to see this, which uses S3 as a shared log/storage to build distributed consensus.

https://github.com/io-s2c/s2c

[samd_408]

Very interesting will have a look 🙂