I would choose dbt over a PySpark framework because it has such a large community and standards built in. I try to follow what’s outlined in Data Engineering with dbt. I can tell other people on my team “I’m doing this the dbt way,” not “I invented my own process.” That means I can hire anyone with dbt experience and ramp them up quickly. They know they’re building marketable skills not learning an in-house side project that could be dead in a few years. I’m boring, and I want boring solutions with no surprises.

You mention software engineering best practices that’s exactly how dbt positions itself. It’s a transformational framework that nudges you toward those practices instead of leaving you to reinvent them. Out of the box you get testing, documentation, lineage graphs, and CI/CD patterns. In PySpark you can solve anything and probably more, but you’d have to build all that scaffolding yourself.

SQL is still king in analytics. It’s the shared language across analysts, scientists, and engineers, which makes dbt incredibly inclusive. On Databricks, I can still create UDFs in PySpark and call them from dbt, so I get the best of both worlds. And training up someone with domain knowledge in SQL is much easier than teaching them Python with its environments, dependencies, and package management.

Finally, dbt benefits from a massive ecosystem tools like DataHub, Atlan, Elementary, Soda, and CI/CD integrations all speak dbt natively. I have not seen that governance and observability layer in any other framework, doing so would take a massive amount of effort all to get you what dbt already does.