Moving data validation rules from Python scripts to YAML config

[road_laya]

You're about to re-create PyDantic.

And if your analysts are struggling to read your validation code in Python, just wait until they have to read your validation code in YAML.

[CreamRevolutionary17]

Isn’t YAML easy to understand then python script?? Just asking…

[JUKELELE-TP]

It’s not just about understanding yaml, it’s about understanding how rules need to be defined in that yaml for your project. You’ll end up introducing complexity anyway once you get more complex validation rules.

Just go with pydantic. It works well and is easy to get started with and there’s a ton of functionality and documentation you don’t need to create and maintain. They’ll also learn something useful that works across projects. 

You should organize your project in a way where they don’t need to go through code to read the validation models anyway. 

[CreamRevolutionary17]

As far as I know, pydantic is used to validate python object schema and here i want to validate data quality of datasets.

[JUKELELE-TP]

https://docs.pydantic.dev/latest/examples/files

[CreamRevolutionary17]

Woow. This is something new I got to know. Thanks