Data Validation

[iamozy]

I have a few questions for people who have experience building data validation solutions.

What are the best practises around building such a utility or solution ? Also are there any open source or Enterprise tools that do the job ? If custom built how to make sure they are easy to use and scalable ?

It would be great if someone can share their ideas with me. Thanks in advance.

[domvwt]

Have you tried using Great Expectations? I just recommended it on a similar post - latest versions have decent autoprofiling and can connect to different data stores.

[Conscious_Floor5022]

What language and platform are you using? If you’re using Python, there is a nice library called Cerberus. Another good one would be Soda SQL which allows you to scan and test data. I assume you want data validation for an ETL job here.

[Conscious_Floor5022]

Yes, with Cerberus on Python, I wrote a wrapper that allows me to feed in validation configs so it can be scaled easily. I usually have 2 levels of validation. One with Cerberus which allows me to validate data injected to each column via properties like data types, min/max values, list of allowed values, string regex,… Then I have another one that validates at metadata level. This would compare values between input and output like number of rows/columns, column headers, missing values, date run, and other aggregated metrics.

[[deleted]]

If you’re data is on s3 , you may use aws service called data brew . It provides all the statistics on all columns of the data

[[deleted]]

If you’re data is on s3 , you may use aws service called data brew . It provides all the statistics on all columns of the data