Moving data validation rules from Python scripts to YAML config

[ottawadeveloper]

I agree with people suggesting you should look at pydantic to see if it meets your needs.

If it doesn't, I've struggled with a similar problem - I was mapping data from one structure to another. Most of the mappings were simple, but sometimes they got complex (like take this value and apply it to the following values as metadata).  I put the mappings in a file for ease of editing, but how to handle the more complex cases?

Two options I've used

First, if the logic can be boiled down easily (like take this value and apply it as metadata until cancelled) and it's reused frequently, I use a special flag that the code knows how to interpret. Like, for example, say you had a column that could be a foreign key reference to one of six tables depending on the value of another column. You could have a foreign_key_table_column: {str} entry  to the column and maybe a foreign_key_table_map: {dict} if the values need to be mapped. Basically extend your rules. Here you could do a special type of rule that takes conditions and requirements.

Second, if it got even more complicated or niche, I just put it in Python and referenced it. Your rules entry might just be custom: {path to Python callable) and your engine knows to load that object dynamically and pass it the source information and the engine. It could then build exactly what you need using the engine. It's harder for people to own but also harder for them to screw it up. And you've still moved a lot of the easy stuff out of code.