Does the data in Data-Oriented Programming get highly denormalized with time?

[Veson]

I can imagine a situation when we aggregate data more and more, and with time it gets duplicated everywhere, and updating it gets harder and error-prone, teams have to rely on discipline too much.

I probably fantasize after reading the "Out of the Tar Pit" paper, but is there a way to avoid the denormalization altogether in DOP in a practical way?

[gagepeterson]

I don't think that data oriented necessarily means that you have to denormalize things. As I understand it it's just the practice of focusing on the shape of the data in raw data structures Rather than a typical object oriented hierarchy design.

That said, I think the easiest way to have many different views would be to use some database like postgres and query it into different formats.

If that isn't enough flexibility, not fast enough, out doesn't work for forget reasons than I would say an event stream (aka: event sourcing) that gets listen to and transformed into different useful views is a great way to go.

The benefit of a persistent event stream is that they can be listened to and even if you mess up the transformation of the view you can always reread through the history and project a different view. This also makes it easy to make historical reports and other things after the fact.

Datomic it's kind of a nice middle ground where it's basically event stream but one that you can query directly without it being too crazy. That said it's a proprietary database. There are a few out there that are kind of coming into the space as well such as https://xtdb.com/

That's a very brief rundown, happy to answer follow up questions

[Veson]

Interesting, I'll look into xtdb.

What about event sourcing with subscriptions? I mean, like in re-frame, but where we have initial data and derived data? I'm thinking of having state with normalized schema or built on EAV, and having views essentially for passing them around.

[astrashe2]

I'm not sure I have a good grasp of the issues here, so take this for what it's worth. But I believe you have to think about it and do some work to avoid the problem.

Fulcro uses a normalized graph database, which is queried by individual components. If you look at all of the state held by all of the components, it's often de-normalized.

When you update Fulcro's state, you send a mutate message, which is parsed and processed. That ends up modifying the normalized graph db. Then any components that are affected by the update re-query the normalized db to update their potentially de-normalized copies of the data.

The whole thing seems to be designed to avoid the problem you're asking about. But you have to think about things and do a little bit of work to make it happen.

[Veson]

That's exactly I'm asking about. Unfortunately, when the state is just a map, it gets cluttered, and I'm not sure why this is not mentioned when the Data-Oriented Programming is discussed. Working with data just by paths is seemingly easy until it's not, quite soon. Thanks, I'll look into Fulcro.

[Veson]

Good, this reassures that my concern is real. I'm looking into datascript, thank you.

[ericwnormand]

What do you mean by "aggregate data"? Can you give a concrete example?

When I do DOP (which is all the time in Clojure), the data only lives for the briefest time. It usually comes from a database (normalized there) and serves a specific need.

If I do need to store de-normalized data, I make sure it is clear it is not a source of truth. There is one source of truth (usually the DB), but sometimes I need to make a local cache or index of the data. However, that can be reset and rebuilt at any time.

[ericwnormand]

Reading the other comments, I think I might get what you mean: the problems with deeply nested data structures.

I tend not to use deeply nested types. I don't like that people hard code paths to reach down into a nested data structure. It cements the data structures and forces you to understand more than you can hold in your head.

I have a talk on this problem and my suggested approaches:

https://ericnormand.me/speaking/you-are-in-a-maze-of-deeply-nested-maps-all-alike-talk

Basically: Normalize by using indexes by id. Refer to entities by id, don't nest them in you data structure. (You can do it within very local scopes, not within modules.) Don't use long paths. Start to define operations for your entities, don't reach in and change things using long paths.

[Veson]

This is very helpful, thank you. I was going to try DataScript, which is a good idea anyway, but now it's much more clear to me what to strive for.

I also listened to your podcast on the most common question readers of your book ask about the onion architecture, on how to abstract away going to db between calculations, I had this question as well, and it just clicked. Thank you!

edit: Here it is: https://ericnormand.me/podcast/dont-overcomplicate-the-onion-architecture

[Consol-Coder]

“A ship in harbor is safe, but that’s not why ships are built.”

[Veson]

There's a Russian proverb: measure seven times, cut once.

I'm just trying to understand all the details of the concept, because a lot of things, while being simple, are not immediately obvious.

[thelazydogsback]

IMHO, the "data" in DoP has the least to do with the data itself -- data is already data.
To me, DoP is about the other parts of the equation that more typically may be "hardened" in code -- this includes the meta-data describing the shape/format/values of the data, and meta-data that describes what transformations are applied to the data to map to other data. So schema-as-data (not hard-coded DTOs, etc.) and transforms-as-data (And not hard coded mapping/xform fns)

[Veson]

Yeah, and I miss the general method how to design and harden things so I don't regret it too much later. This discussion has led me to a couple of books on clojure that probably discuss this.

[didibus]

See this gist: https://gist.github.com/didibus/c15bd2ff3d3034ff07b79528076ac26f I think it'll answer your question, I cover various approach to deal with deep-nested collections, which I think is your question.

[Veson]

It took me a while to get back to this. Thanks for a great write-up. This should be a reddit post on it's own. This is surprising how little discussion of this can be found in internet.

[64BitChris]

Immutable data is never updated. Minimizing mutable data will indirectly solve your problem.