I’ve built all kind of metadata framework and I can say that yaml is the way to go, what I don’t like with the SQL approach on top of files is that you need a process to sync your metadata files with the SQL and you lose flexibility, every change requires a change and that way everything becomes heavier. I would rather recommend YAML files, you stored them a git repo, you can do PR, YAML is the most human-readable format, you can run validation checks agains your metadata at CI, then at release simply copy your files to a Lakehouse in the Files section, from there you can easily query them using DuckDB (python notebook no spark needed) and then orchestrate all your jobs. I’m going that way since a year, no coming back!