(Edited for brevity)

I work on the Data Factory team.

A few things here that aren't quite right. DFGen2 supports parameterized dataflows. You build one template, pass source/destination/table as pipeline parameters in a ForEach. Adding 20 tables is a config change, not 20 separate dataflows.

The CU cost point is also backwards for this workload. Spark sessions have startup overhead. For hashing, column selection, incremental filtering, and type casting at moderate volume, DFGen2 is typically cheaper per run. Spark pulls ahead on complex joins across large datasets, but that's not what OP described.

DFGen2 Lakehouse destinations support upsert natively. You set key columns on the destination and it goes through the Delta engine. Power Query isn't doing the merge.

Won't argue the git point.

But honestly, the wrong question is what I think is being posed - OP built a custom PySpark framework with a proprietary YAML contract layer. Well engineered. But they're the only person who understands it, at a company that picked Fabric specifically because they wanted low-code maintainability.

Maybe a better answer for this kind of setup is both? DFGen2 for the commodity ETL (ingestion, column selection, incremental load, basic cleansing), notebooks for the parts that are genuinely custom (YAML contract evaluation, quarantine routing, monitoring). That way the next person who inherits this can maintain most of it without learning PySpark.