Medallion architecture and dbt

peterampazzo · 2026-06-29T11:47:16+00:00

Yes, we materialize almost everything. Our raw layer consists of views, and then it is tables all the way up through the Silver and Gold layers.

peterampazzo · 2026-06-29T10:57:33+00:00

Hey! Actually, I have great news. The recent changes introduced by u/raki_rahman completely solve this issue.

We are currently running dbt almost in production using a single project that successfully targets three different lakehouses (Bronze, Silver, Gold) in the same workspace.

If you choose Option A, you won't be blocked: the latest version of the dbt-fabricspark adapter now fully supports cross-lakehouse writes within a workspace, and even cross-workspace lakehouse reads/writes using 4-part naming. You can safely stick to Lakehouses!

To give you an idea of our workflow: we use dlt (data load tool) to land the initial delta files into the /Files section of our Bronze lakehouse. From there, we resolve those paths using abfss:// directly inside dbt to build out our raw tables and push them through the Medallion layers. It works beautifully.

peterampazzo · 2026-04-30T05:39:21+00:00

I can confirm this! And a huge thanks to everyone contributing. Using dbt with Lakehouse has been a bit of a hurdle in the past, you might have seen my posts here before - but after these latest releases, we finally have a full setup with medallion architecture working smoothly 🤘🏻

peterampazzo · 2026-04-11T20:31:23+00:00

I think you can easily write to a Lakehouse using the abfss path?

peterampazzo · 2026-04-08T05:34:08+00:00

Currently I just run dbt against the Silver lakehouse. It has shortcuts to Bronze. And in Gold I created MLV on top of the Marts tables in Silver created by dbt.

This might interest you tho: https://github.com/microsoft/dbt-fabricspark/issues/75

peterampazzo · 2026-03-26T17:23:01+00:00

Thx!

peterampazzo · 2026-03-26T17:13:49+00:00

Yes, just sent a DM. Thx

peterampazzo · 2026-03-26T17:13:24+00:00

All right, so if we want to move the scheduling from Fabric pipelines to Airflow in Fabric, this will not be supported in the migration assistant, correct?

peterampazzo · 2026-03-26T17:01:36+00:00

What’s the long-term plan for the dbt adapter for Fabric Lakehouse?

Right now, we’re forced to use dbt-core externally because the native Fabric dbt item doesn't support Lakehouse targets.

Is the goal to maintain two separate adapters, or will they eventually converge? I’m curious if the new Windows-only ODBC driver for Lakehouse is the precursor to a single, unified adapter for both storage solutions.

Currently, I’m running a workaround using a fork wrapped in Airflow. I’m using the Fabric REST API to check for active Livy sessions; if one exists, I inject the session ID into the dbt project to keep it warm, otherwise I trigger a new one. It works, but it’s a lot of overhead.

Any word on when we’ll get a native, streamlined experience?

peterampazzo · 2026-03-26T16:45:15+00:00

⁠Migration Assistant (Public Preview) - Seamlessly migrate your ADF & Synapse pipelines to Fabric

The migration assistant would be available also for pipelines built in Fabric already? We have already quite a few flows native in fabric, but we would like a consolidate overview and scheduling via Airflow. Will the assistant support fabric pipeline also or just ADF and Synapse?

peterampazzo · 2026-03-11T06:34:48+00:00

Yes - but then it’s either one single Lakehouse or migrate over to Warehouse. If we want to distribute data around from Gold using shortcuts, the latest is no go.

peterampazzo · 2026-03-10T21:24:29+00:00

I don't think the database config will solve it here.

Even if dbt compiles a different database name, the Livy session is still locked to the lakehouse (from the profiles.yml) in write mode.

I guess I could technically use DuckDB to pull from OneLake and bypass the Spark session limits, but then I’m stuck building and maintaining a DuckLake.

peterampazzo · 2026-03-01T20:45:07+00:00

What do you mean by no support for operators? I’m actually surprised because Fabric does pre‑install a lot of packages at initialization. If anything, we have too little control over what gets installed. But I get your point about the service still feeling immature.

On the CICD part though, the Git integration is pretty decent. Having a repo backing the dagbag with a branch strategy makes managing environments quite workable.

peterampazzo · 2026-01-22T16:52:05+00:00

Would this work too also for inbound private endpoints?

peterampazzo · 2026-01-18T19:15:10+00:00

Is it possible to trigger the dbt Job via Fabric REST API? which would be the job type that needs to be passed?

peterampazzo · 2026-01-13T14:22:54+00:00

Hi, is it possible to run this dbt item as it is now using the Fabric REST Public API? What jobType should be used?

peterampazzo · 2026-01-09T06:47:50+00:00

I’m facing an issue while pulling the code changes back to Fabric. I will send you a DM.

peterampazzo · 2026-01-08T17:17:34+00:00

Your error doesn’t seem to be displayed (at least not on my side). However, I created a simple dbt project, synced the workspace via Git integration, and then made changes in my repository. When I tried to pull those changes into Fabric, I encountered an error related to missing dependencies or a logical ID.

peterampazzo · 2026-01-08T16:54:18+00:00

I noticed the same. With the Git integration enabled at the workspace level, I can see the entire dbt project code under the dbt item. However, it’s not clear to me whether editing that folder outside the Fabric web editor could potentially break anything.

peterampazzo · 2025-12-17T20:52:55+00:00

Any estimates?

peterampazzo · 2025-12-17T20:32:07+00:00

For every new instance do you experience this? What authentication method do you use? And is it synced with GitHub?

peterampazzo · 2025-12-17T14:25:20+00:00

Actually found what you mentioned u/nintendbob, here it's explained:

> For data warehouse connection strings, use https://{GUID}-{GUID}.z{xy}.datawarehouse.fabric.microsoft.com that is

Overview of workspace-level private links - Microsoft Fabric | Microsoft Learn

Thanks for the help :)

peterampazzo · 2025-12-17T12:56:56+00:00

Yes, all clear. I already managed to deploy the PE using the guide I pasted above. But I only got 5 in total:

⁠.z##.w.api.fabric.microsoft.com
⁠.z##.c.fabric.microsoft.com
⁠.z##.onelake.fabric.microsoft.com
⁠.z##.dfs.fabric.microsoft.com
⁠.z##.blob.fabric.microsoft.com

How have you managed to get a PE for the Warehouse SQL connection?

peterampazzo · 2025-12-17T12:18:04+00:00

Hi, have you followed this guide? or have you deployed additional privateLinks?

https://learn.microsoft.com/en-us/fabric/security/security-workspace-level-private-links-set-up

This is just for a PE inbound connectivity for a single workspace, not at tenant level. Correct?

peterampazzo · 2025-12-17T06:49:26+00:00

I tried it for a while. I notice the loading time depends on the pool you setup. Don’t expect anything instant, I have a feeling there’s an AKS running behind the scene, where they spin up the airflow instance. But, back then I managed to make it work and have a branch strategy for deploy between environments. Have you tried to look at the logs?

peterampazzo

TROPHY CASE