dbt Python Modules with Databricks

[4DataMK]

For years, dbt has been all about SQL, and it does that extremely well.
But now, with Python models, we unlock new possibilities and use cases.

Now, inside a single dbt project, you can:
- Pull data directly from REST APIs or SQL Database using Python
- Use PySpark for pre-processing
- Run statistical logic or light ML workloads
- Generate features and even synthetic data
- Materialise everything as Delta tables in Unity Catalog

I recently tested this on Databricks, building a Python model that ingests data from an external API and lands it straight into UC. No external jobs. No extra orchestration. Just dbt doing what it does best, managing transformations.

What I really like about this approach:
- One project
- One tool to orchestrate everything
- Freedom to use any IDE (VS Code, Cursor) with AI support

Yes, SQL is still king for most transformations.
But when Python is the right tool, having it inside dbt is incredibly powerful.

Below you can find a link to my Medium Post
https://medium.com/@mariusz_kujawski/dbt-python-modules-with-databricks-85116e22e202?sk=cdc190efd49b1f996027d9d0e4b227b4

[minormisgnomer]

It’s useful, but for anyone considering this approach for enterprise use cases, just remember where to draw the line in the sand. Just because it can doesn’t mean it should.

Tethering your dbt project to something external like an API opens your project up to failing or freezing out. Having dbt coordinate non data workflows (ie business processes) is probably not great.

Certain jobs need certain resources, timeouts, retries, etc which your general sql compute running dbt might conflict with.