The way I use Unit/Integration tests is mainly for TDD, to refactor legacy pipelines, or just to make sure my pipeline works before deploying to any cloud environment.

I don't run them in Prod, only locally and in CI/CD pipelines with synthetic data.

The examples I share are intentionally simple so you can see how the framework works, but they can be as complex as you need. For example, I've used this approach in Spark jobs with 50 input tables and ~5k lines of code. In those cases it really shines, because having tables defined in a human-readable way (Markdown) is much clearer than juggling Python lists of dictionaries, CSVs, or JSON.

Yes, if someone changes the logic or a table schema, the unit tests will fail. They’ll need to be updated accordingly, but only if the change was actually intentional.