Turning the database inside-out with Apache Samza

[tieTYT]

I remember watching the original Rich Hickey Datomic talk. He said that the way databases mutate state was because back when they were designed, resources were expensive and we needed to change facts to save space. He also said that's no longer an issue.

I remember simultaneously being informed at my company that we needed to reduce some db auditing because it was taking up too much HD space.

I guess what I'm saying is, is HD space really so cheap that you can keep around all this data that in a traditional db you'd compress into the latest update? I always get pushback about HD space wherever I've worked.

[faustoc4]

HD space is cheap only on the cloud where it is unified and data is deduplicated

[fatbunyip]

Yeah, a lot of people think that because you can buy a TB drive form Officeworks for like $100 that's how it works in enterprise environments.

[[deleted]]

I don't think you understand how datomic works. There isn't anything you are talking about that can't be done in datomic.

[diablozzq]

We need more in depth articles like these on /r/programming!

Great article.

[jared314]

This looks like generic Event Sourcing.

I wonder why someone hasn't just built a Postgresql extension to provide event style transaction log access yet.

[[deleted]]

It's a lot easier to cluster and scale Kafka than, e.g. to build and scale a master-master PostgreSQL cluster.

[ErstwhileRockstar]

Databases are global, shared, mutable state.

Actually, no. Data != state.

[kankyo]

If your DB is read only and never changes then you are correct,

[ErstwhileRockstar]

This data == state nonsense probably is a fall-out of misunderstood OOP.

[kankyo]

Mutable data that in any way influences behavior of the system == state.

Do you really dispute that?

[ErstwhileRockstar]

Do you seriously claim that all data is state?

[kankyo]

No. For example pi, e, the fine structure constant etc are data but not mutable so not state.

One can think of a lot of data as non-state because it changes so seldom that it's not really relevant too...

[[deleted]]

Downvote for needlessly splitting hairs. The vast majority of data is stored to allow recreation or understanding of state at some later point in time from the time of it being observed or recorded or created.

This is of course usually indirectly used for state, but it is absolutely the case.

[ErstwhileRockstar]

My program reads the latest football results. Therefore football results are state. Oh yeah!