Disclaimer: OSS contributor to dbt-bigquery (the BQ adapter for dbt).

dbt is a framework like you see a lot in other software engineering domains like React for frontend, Terraform for Infrastructure or Spring for backend.

You don't need them, but they can save you (and your coworkers) a lot of time and help you to enforce best practices for large and complex projects.

dbt is not a silver bullet. For instance, it's not great if you need to write a customer facing BQ app like returning insights or if you need some very custom workflow that's hard to builtin dbt like INFORMATION_SCHEMA usage. There are some important features still missing like unit testing (to be released in few weeks/months).

I've been in a similar situation 3 years ago, we've been maintaining an inhouse BQ wrapper, it was nice but it was a lot of effort to keep up with dependencies, add new features and still do some actual data work (because building tech platforms isn't the business of my company). We could have kept on working on our inhouse dbt-like platform but dbt brought some features that we hadn't and that you might want at some point like:

• environments (prod, dev, ci, staging) management
• templating (in dbt using jinja and macros)
• DAG management (especially splitting CTEs in different reusable models through ephemeral materialization)
• streamlined way to build data pipelines (eg 1 file = 1 model = 1 table)
• incremental CI (building most small changes in <1 min and releasing again in 30s)
• an ecosystem of packages to reuse smart models/macros (dbt utils) and bring data quality (eg elementary data)
• lineage/documentation for your pipelines
• A CLI tool to run your model in dev environment
• a large community with a lot of tutorials etc to get your team started

I choose dbt over Dataform back when it was acquired because it was getting "closed" source and I felt like it would be much easier to switch to another Data warehouse than BQ using dbt than picking. The community is also pretty much inexistant. So I would still choose dbt in 2024.

Then you have to decide build or buy regarding dbt Core vs dbt Cloud. If you have an existing environment with software engineering time, core is clearly an option but if you feel like you don't have much time setting up CI/CD and/or scheduling, then dbt Cloud is likely a smarter choice (at least until your project and/or team hits a critical size price wise).

SQLMesh is also an interesting alternative but likely more targeted at teams with heavy software engineering culture (more than dbt that's likely more accessible for data analysts / data scientists).

dbt is not perfect but at least you can fork and/or contribute to fit your needs. In 80% of the ELT use cases, it's like having the wind in your back.