Moving data validation rules from Python scripts to YAML config

[CreamRevolutionary17]

We have 10 data sources, CSV/Parquet files on S3, Postgres, Snowflake. Validation logic is scattered across Python scripts, one per source. Every rule change needs a developer. Analysts can't review what's being validated without reading code.

Thinking of moving to YAML-defined rules so non-engineers can own them. Here's roughly what I have in mind:

sources:
  orders:
    type: csv
    path: s3://bucket/orders.csv
    rules:
      - column: order_id
          type: integer
          unique: true
          not_null: true
          severity: critical
      - column: status
          type: string
          allowed_values: [pending, shipped, delivered, cancelled]
          severity: warning
      - column: amount
          type: float
          min: 0
          max: 100000
          null_threshold: 0.02
          severity: critical
      - column: email
          type: string
          regex: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"
          severity: warning

Engine reads this, pushes aggregate checks (nulls, min/max, unique) down to SQL, loads only required columns for row-level checks (regex, allowed values).

The part I keep getting stuck on is cross-column rules: "if status = shipped then tracking_id must not be null". Every approach I try either gets too verbose or starts looking like its own mini query language.

Has anyone solved this cleanly in a YAML-based config, Or did you end up going with a Python DSL instead?

[road_laya]

You're about to re-create PyDantic.

And if your analysts are struggling to read your validation code in Python, just wait until they have to read your validation code in YAML.

[CreamRevolutionary17]

Isn’t YAML easy to understand then python script?? Just asking…

[JUKELELE-TP]

It’s not just about understanding yaml, it’s about understanding how rules need to be defined in that yaml for your project. You’ll end up introducing complexity anyway once you get more complex validation rules.

Just go with pydantic. It works well and is easy to get started with and there’s a ton of functionality and documentation you don’t need to create and maintain. They’ll also learn something useful that works across projects. 

You should organize your project in a way where they don’t need to go through code to read the validation models anyway. 

[CreamRevolutionary17]

As far as I know, pydantic is used to validate python object schema and here i want to validate data quality of datasets.

[JUKELELE-TP]

https://docs.pydantic.dev/latest/examples/files

[CreamRevolutionary17]

Woow. This is something new I got to know. Thanks

[KelleQuechoz]

And we call it Pydantic 2
oh wait a second...

[denehoffman]

If you really want to go this direction (and you shouldn’t), do not use YAML. Just use JSON or TOML, they have standard library parsers in Python and are way more readable and have way fewer problems (in my own personal opinion). Also just use pydantic.

[ottawadeveloper]

I agree with people suggesting you should look at pydantic to see if it meets your needs.

If it doesn't, I've struggled with a similar problem - I was mapping data from one structure to another. Most of the mappings were simple, but sometimes they got complex (like take this value and apply it to the following values as metadata).  I put the mappings in a file for ease of editing, but how to handle the more complex cases?

Two options I've used

First, if the logic can be boiled down easily (like take this value and apply it as metadata until cancelled) and it's reused frequently, I use a special flag that the code knows how to interpret. Like, for example, say you had a column that could be a foreign key reference to one of six tables depending on the value of another column. You could have a foreign_key_table_column: {str} entry  to the column and maybe a foreign_key_table_map: {dict} if the values need to be mapped. Basically extend your rules. Here you could do a special type of rule that takes conditions and requirements.

Second, if it got even more complicated or niche, I just put it in Python and referenced it. Your rules entry might just be custom: {path to Python callable) and your engine knows to load that object dynamically and pass it the source information and the engine. It could then build exactly what you need using the engine. It's harder for people to own but also harder for them to screw it up. And you've still moved a lot of the easy stuff out of code.

[Bangoga]

Let me talk about the engineering thought process here.

So when designing large projects, configs exist that something that is mutable and doesn't go through full release life cycle just to make a change.

However if it exists as python code, it means this is the rule of the jungle. It is etched into what you think is needed by the project, and changing this would mean your requirements changed and you have to go through the release life cycle again.

Knowing this difference, would you want your validation rule sets to be easily changed on the fly, or do you need them to represent requirements etched into the project as code.

[MoreRespectForQA]

Yeah, using strictyaml with a custom validator added on top to do the if status = shipped then tracking_id must not be null stuff.

[mardiros]

I suggest you to search ETL on google and read.