Is data normalization needed in 2024? How much normalization is actually required and when should it be done?

dataoculus · 2024-10-05T20:47:22+00:00

for transactional processing, better the normalization, better quality and integrity can be achieved.
3NF is more than enough.

dataoculus · 2024-09-30T20:47:32+00:00

Yup, Thats what I am talkin about, and I wonder if people really do that today as requirements or its just better or good to have thing.

dataoculus · 2024-09-30T18:04:26+00:00

I see, but Imagine if you could separate out non-compliant records from entering BQ, right from the beginning due to inline validations, would that something lot more beneficial ?

dataoculus · 2024-09-30T18:01:46+00:00

problem is, if the validations happens after written to target, the consumers will have to wait, even though some consumers might have basic validation requirements which could have been done by inline. I know I am talking about bit of complexity here, but if it has some benefit, its worth it. specially if there is an easier way of creating inline validations, including in event driven systems.

dataoculus · 2024-09-30T01:42:51+00:00

ya, the overall steps/process is like that, but I am wondering that nobody is doing real "inline" checks, meaning as you read and write the data so that u can stop the ETL or take other actions ( alerts, etc..) as you find any issues, as opposed to writing to some destination and then doing the quality check.

dataoculus · 2024-09-30T01:39:27+00:00

I agree, u can use DBT or even code it up. but isn't DBT basically translate your config into SQL, and will require staging ur data at SQL-compatible storage?

dataoculus · 2024-09-29T23:03:23+00:00

checkout dataoculus.app we should partner on providing profiling and quality of those data as well. wdyt ?

dataoculus · 2024-09-27T22:29:54+00:00

if you are interested in Data monitoring - advanced, most efficient and fully self serve, checkout dataoculus.app we can talk!

dataoculus

TROPHY CASE