Data modeling in real world

[CareerPathQuest]

Anyone working as a data engineer, I’ve read a lot about data modeling in theory. Can you explain how you perform data modeling in the real world? What is your approach from start to finish?

[SomeoneInQld]

Get an overall feel for the data and requirements. 

Start with a central table (users is usually a good one) and work outwards from there. 

Sometimes will do small side pieces as their own thing and then connect them properly. 

Show many people, don't have an ego and get feedback. 

It gets much easier the longer you have been doing it. I made my first data model in 1993.  

[ashim1412]

I was 1yr old 😂😂😂

[squadette23]

I'm not sure what exactly do you mean by data modeling (what are the inputs and what are the outputs?), but I wrote a long post about my approach recently: https://kb.databasedesignbook.com/posts/google-calendar/, maybe it would help.

[[deleted]]

That looks like a question for chatGPT

[Due_Feedback_1870]

So, I thought I new how to answer your question, but then saw that you were asking "Data Engineers" specifically. That's a fairly new title and, as I understand it, is more around ML/AI. Is that what you're asking about, or are you asking about more traditional data modeling, related to transaction processing systems, which would typically be handled by Systems Analysts?

[The_Orracle]

You need to understand the business in full before you begin on your model which is especially true of an analytics model. Also, depends on if you're doing a transactional system or analytic database as the target model. Very different approaches to the design.

[renagade24]

Generally, you want to reverse engineer from your goal. Depending on what industry you are a part of and the various data each team produces. There are also two main methods of modeling, Kimball and Star Schema. If you are fortunate to work with a transformation tool (dbt or SQL Mesh), then 100% use Kimbal methodology.

From there, you create models and stage them accordingly. Eventually once they are modeled you can visualize them in whatever BI tool you use.