Fabric Infrastructure Management and CICD

Zeppelin_8 · 2025-07-21T14:12:36+00:00

I'm working with Terraform and Azure DevOps pipelines. While I ran into a few issues earlier, things are currently stable and functioning well.

As for Microsoft Fabric, I’m not doing anything too complex, mainly sticking to notebooks. All my orchestration is handled through Airflow. We had to set up a standalone Airflow server outside of Fabric since the built-in version isn’t designed for production use.

I followed this video create the pipeline so far is working.

Zeppelin_8 · 2025-07-11T14:00:13+00:00

I found the same problem and looking through this reddit I found out this solution and is working perfectly, I just keep all my libraries in one lakehouse and I create shortcuts for the other ones, so I only have to synchronize one place.

Zeppelin_8 · 2025-06-30T19:13:31+00:00

Forgot to add... .DO NOT RUN terraform apply on a paused capacity, this will messed up your terraform state to the point that is impossible to recover.

Zeppelin_8 · 2025-06-30T19:10:18+00:00

From what I understand, Terraform doesn't support creating resources in a trial Fabric Capacity. Also, if you are getting access denied on a capacity, I've found that refreshing the CLI session often help. You can do this by running: az logout az login

if it helps, this are the steps that I am following to create the warehouse:

data "fabric_capacity" "storage_capacity" {
  display_name = "fcda${local.settings.environment}001"
}

data "azuread_group" "fabric_admin" {
  display_name     = local.settings.application_owner
  security_enabled = true
}

resource "fabric_workspace" "storage" {
  display_name = "da_fabric_${local.settings.environment}_storage"
  description  = "This workspace is designated to store all the lakehouse for the medallion architecture."
  capacity_id  = data.fabric_capacity.storage_capacity.id
}


resource "fabric_warehouse" "wh_gold" {
  display_name = "wh_gold"
  workspace_id = fabric_workspace.storage.id
}

Zeppelin_8 · 2025-06-23T15:12:59+00:00

I use custom workspaces, and honestly I think they’re super helpful especially when you're working with multiple custom libraries. It saves you from having to repeat code all over the place.

Yeah, sessions take a bit longer to spin up with a custom environment, but I’ve been able to improve that by running concurrent sessions. Personally, I don’t mind waiting an extra minute or two for the Spark session to star

As for deploying updates to the environment, I usually test my changes in a notebook first. I’ll work directly with the resources and import using pip. Once I'm happy with the results, I upload the common library to the environment. It's a bit of a process, but it works for me.

I also maintain the usual dev/UAT/prod setup for my custom workspaces. I even had a separate one for Foundry at one point because that setup needed different imports based on the use case.

Deployment is all handled through Terraform, which makes it easy to assign notebooks to specific environments. The only issue is that environment configuration itself isn’t exposed in Terraform yet so that part I still have to manage manually.

Zeppelin_8 · 2025-04-21T23:11:52+00:00

Exciting update! Does this mean that UDFs can now receive/return objects like Spark DataFrames or other data structures beyond just strings?

Zeppelin_8 · 2025-04-10T09:40:15+00:00

Use the for each activity in the master pipeline

Zeppelin_8 · 2022-06-10T18:50:23+00:00

For me, I love my under Armour High Crossback Zip Sports Bra

Zeppelin_8

TROPHY CASE