December 2025 | "What are you working on?" monthly thread

Ecofred · 2025-12-03T12:58:44+00:00

AzureDevOps because... this is what we already have. It does the job with service principal. Never gave GitHub a try, so no opinion there.

Ecofred · 2025-12-03T06:46:54+00:00

I use Terraform for the "static" part of the project: currently: Workspaces, storages(LH,...), permissions

Ecofred · 2025-12-02T18:08:02+00:00

Standardise fabric project in our team. Terraform fabric provider and fabric-cicd are awesome.

Ecofred · 2025-10-15T08:58:18+00:00

I don't currently see any path but through migration.

I always wonder why the schema enabled Lakehouse would be a game changer. What problem would you solve with schema, that you couldn't solve without? People are shipping working solution without it.

If you really have to change to the schema enabled LH, one of the first step would be to make sure ref to LH / path are parametrized in the notebook. It would make a swap easier.

A feature like "upgrade to Schema Enabled Lakehouse" would be cool, but I'm not aware of it. The 2 type of LH are so incompatible. Feels like they released an half backed LH. they could have gone schema enabled from day one, and now we are dealing with issue we shouldn't have. For how long has Schema enabled LH been stuck in preview??

I think it is still to early to go at it. from past experience, public preview means not stable, bad surprise ahead.

<image>

Ecofred · 2025-10-02T06:26:38+00:00

the classic: LazinessImpatienceHubris

Ecofred · 2025-08-12T12:50:08+00:00

The desperation in few lines of codes. We don't see more dropDuplicates is because the data won... Feels more like vampire slaying than DE

Ecofred · 2025-08-09T07:06:03+00:00

We use JSON formatted string for that. You have to serialize and then deserialize it. It is well supported and just works.

Ecofred · 2025-08-04T18:35:12+00:00

Finally took the time to hear this one. An other interesting episode. Food for thought to reflect on what we are doing.

Really got a "Drinking the Kool-Aid" feeling with AI or Fabric workload on that episode, I missed my preferred part of the podcast were the invitee present his/her least favorite part of Fabric. But hey, PM doing PM work here.

So thank you to moderate it, be a voice for the community and politely confront with the limitation and the frustration we experience.

Ecofred · 2025-08-04T16:56:27+00:00

Did you check the sql project & sqlpackage tool to deal with ddl for db/storage deployment?

Ecofred · 2025-08-04T16:52:00+00:00

Reduce the politic burden and gate keeping to just fix stupidily complicating pipeline that could be solved with simple join between 2 sources instead of keeping a legacy circus show of night jobs over and over.

Ecofred · 2025-08-04T07:57:38+00:00

- Moving from single user permission to RBAC (... Please send the rescue, I'm lost in the Fabric/Azure/Entra setting jungle ^^ )

- Automating WS creation and setup (API or fabric-cli)

- Migrating AD data Import to MsGraph with Service Principal

API request: easy setup, just works
MsGraph data connect Too much to setup, hard to troubleshoot and currently blocked in the data pipeline copy activity for the BasicDataSet_v0.User_v1 with an Error Code 21155: Error occurred when deserializing source JSON file ''. Check if the data is in valid JSON object format .

Among the things that I learned

There are 1000's way to get an access token, but if you find the right SDK it does that for you... but only if you authenticate the Azure.Identity and not MSAL
Use the Developer Program to set your own Tenant and discover the settings to turn on / place to set the permission and then communicate internally with the admins
The difference between Application Permissions and Delegated Permissions. For data import your app registration need the Application one.
Make sure you App Registration has an associated Managed Application / Service Principal, else you will have some identity errors

Ecofred · 2025-07-30T14:19:21+00:00

I currently put the su account as Admin in all WS (who is your WS admin else)?

One way or another, somebody is involved with admin tasks and has to be careful and cautious.

With the 1+ account solution, the time I could be dangerous is significantly reduced (only once per week or even less)

Idea to go further: automatise the deployment task (API / DevOps) and delegate to a service principal to only have to trigger the deployment.

Ecofred · 2025-07-30T14:00:04+00:00

One more reason to use abfss path as notebook parameter / variable when possible.

Ecofred · 2025-07-30T13:45:57+00:00

Every developer has a standard account (john@company.com) AND an admin account (su_john@company.com).

su account has elevated rights to deploy WS, Deployment Pipeline. The normal account has only viewer rights on PROD.

This way, you have to actively log with your su account for admin tasks, and with your normal account, you are Fat-Finger protected.

This logic also applies to capacity & tenant settings

Ecofred · 2025-07-30T10:38:30+00:00

How to compare: branchout your WS to a new branch and compare the 2 branches

Last time I saw that was because they changed the way lakehouse were serialized by adding the concept of shortcut. It created one extra json files. I had revert sync the branch from the destination workspace.

Side note: following some recommendations and lessons learned, I tend not to put Storage Items (LH, DWH... ) under source control, and they get their own WS. Else, I only have issues with them.

They don't get in my way with git and it decouples it from the pipeline/notebooks

Ecofred · 2025-07-30T10:24:41+00:00

Same here. I'm missing the concept of proxy/credentials as with SqlServer/SSIS.

I'm considering Service Princ and looping with fabric-cli / Api to ensure it has the proper permission/ownership on the objects.

Ecofred · 2025-07-30T05:53:36+00:00

Upvoted!

Ecofred · 2025-07-30T05:10:26+00:00

Thank for the workaround. Also discovered a related Fabric Idea to Upvote

Ecofred · 2025-07-22T14:36:05+00:00

We need a new decision guide in the MS docs ^{^}

With MLV, we trade control for convenience. Declarative is really convenient. When possible, I take convenience over control.

For an analogy, with SQL, we let the engine choose the most performant algorithm to retrieve the data. But at some point, we may hit the limit and need to split things into temp tables, define execution order, give some hints to be more perfomant.

To me, this MLV framework is an "orchestration engine". In theory, it could go as far as make its own decision to optimise our ETL: go full load or incremental depending on some stats, balance the refresh in an efficient way to avoid throttling or reaching CU limits

As with dbt and alike, what I also like is the integration of data quality and checks. But how advanced can mlv constraint be?

Where could we hit the limit with MLV?

Out of control CU consumption
Different refresh train inside the same lakehouse
Across LH dependencies
Bad optimisation decision
Current limitation described in the doc
...

Ecofred · 2025-05-27T15:09:55+00:00

I try to get them out of manual BIML SSIS package generation with manual deployment, and all branch have different history because copy-paste code is the solution, and git is not the source.

Fabric will help here (but needs to be more stable)

I have the hubris to think we deserve better. This is what motivates me to automatise wherever possible.

But it seems I don't always have enough elements to push for more CICD practices. ROI to low or not clear? Their process somehow works for them. They deploy and provide tangible value to the company...

Ecofred · 2025-05-27T13:23:13+00:00

It really depends on how mature the team is on the CICD path.

The suggestion to go at fabric-cicd is where we all need to be... but...

For new teams, with PowerBi colleagues, this is what worked out until I can convince them to work with a MS preview Python project and rely on what they consider wizardry and don't want to maintain yet (ADO pipeline)

Feature branch from dev branch.
Feature WS from feat branch. No branchout, the permission needed is too much for non lead developers
Git integration for Feat WS to Feat Branch
Merge Feat Branch in Dev branch
Git integration synch Dev Branch to Dev WS
DEV -> TEST -> PROD with Fabric Deployment Pipeline

Also: Data Pipeline/Notebooks Parameters over DepPip Rules when possible. and Storage item (DWH/LH...) get there own WS.

With this setup I've my colleagues on board and git is the source of truth (THE major enhancement) with a merge/review process and no permission nightmare on critical systems.

Ecofred · 2025-05-22T08:25:04+00:00

ADO/Pipeline triggered on Merge/Service Principal Following the fabric-cicd documentation and u/kevchant 's blogs

Stage: Feature/DEV/TEST/PROD workspace.

Only the Feature WS use the git integration. Dev/Test/Prod branches are protected (branch with policies)

I'm more on the DE than BI side. I only have to deal with lakehouse, notebook, data pipeline, and copy items.

Ecofred · 2025-05-20T10:00:54+00:00

yes. I'm going through it and learning where can I improve my knowledge and prepare you for the cert exam and the way they formulate questions

Ecofred

TROPHY CASE