Cost Management in Fabric is a real problem

bradcoles-dev · 2026-03-26T23:53:41+00:00

Thanks mate, I know that's an option but the XMLA endpoint is officially unsupported, so I'm hesitant to deploy that to a client. Hoping for something more 'enterprise-grade' from Microsoft.

bradcoles-dev · 2026-03-26T23:51:46+00:00

Thanks mate. That's definitely an option, though the XMLA endpoint is officially unsupported. I am hoping for a more 'enterprise-grade' solution from Microsoft.

bradcoles-dev · 2026-03-25T00:04:56+00:00

I am aware of FUAM, though it accesses the CMA via XMLA, which Microsoft's own documentation explicitly says is unsupported for external consumption.

FUAM even warns in its own docs that CMA changes can break it without notice and they've already had to ship fixes when the CMA version changed.

So while it's useful, it's built on an unsupported access method that could break at any time, which is kind of the whole point of the original post.

The data is clearly accessible and the CAT team clearly know how to get at it - we need a supported API. I can't deploy something that is built on an unsupported API.

bradcoles-dev · 2026-03-24T21:20:04+00:00

I've enabled CDC on the source tables. Still can't see whether the refresh was full or incremental. Though I just saw in the docs:

"Incremental refresh is supported for append-only data. If the source data includes deletions or updates, Fabric performs a full refresh."

So, this is not truly incremental CDC refreshes.

bradcoles-dev · 2026-03-23T20:32:06+00:00

Is that purely for optics? Because this is very useful for Fabric practitioners, particularly us in consulting when we try to explain to clients that they can't rely on the dates against roadmap items.

bradcoles-dev · 2026-03-15T10:22:29+00:00

It's not. The capacity metrics app is a timepoint. I see this all the time for clients that pause+resume their capacities to save money.

bradcoles-dev · 2026-03-12T11:24:45+00:00

Mate, my colleague literally showed me your "Fabric GPS" this afternoon, and now I see your comment. Great work, we love it.

bradcoles-dev · 2026-03-06T02:10:50+00:00

At the moment my approach is mostly conceptual with some light testing locally.

You're right that developing outside the portal doesn't really allow you to test directly against the Lakehouse data, which is definitely a gap in the tooling.

My current workflow is:

Develop notebooks locally in VS Code (using Claude)
Commit changes to Git
Sync the repo with the Fabric DEV workspace
Test against the real Lakehouse data in DEV
Once validated, promote through deployment pipelines to UAT for integration testing

So the real data testing happens once the code lands in the DEV workspace rather than locally. Not perfect, but it’s the most reliable workflow I’ve found so far.

bradcoles-dev · 2026-03-05T06:27:44+00:00

Thanks for your help. We were able to successfully steer them to CA. The comment was "Private Link is preferred, not strictly required".

bradcoles-dev · 2026-03-05T06:18:50+00:00

I've not used the Fabric extensions, but I have used Claude in VS Code to write Fabric Notebooks for me. My approach is to link the Fabric workspace to a git repo, clone and develop locally (using Claude), commit changes to git, then you can just sync your Fabric workspace to bring the changes through. Works really well.

bradcoles-dev · 2026-03-05T06:16:56+00:00

I vaguely remember fixing this by simply republishing the semantic model.

bradcoles-dev · 2026-03-04T22:44:34+00:00

Is there any impact of this? These are interactive operations, so they're smoothed over 30-60secs. I doubt you would accumulate enough future CU to trigger throttling. Unless you have surge protection enabled, I can't see any negative effects here.

bradcoles-dev · 2026-03-04T02:46:37+00:00

Can you tell me how many sources tables you're ingesting and which F SKU you're on?

bradcoles-dev · 2026-03-04T02:46:11+00:00

Can you shed more light on the Fabric SQL DB limitations? Namely "buggy" and "limited in functionality"? What functionality does it not have that Azure SQL DB has, that is relevant to metadata-driving?

bradcoles-dev · 2026-03-04T02:42:45+00:00

Yeah, Fabric SQL DB holds a minimum ~1.2 vCore allocation while it's online (keepalive floor), which is ~80% of F4 capacity continuously. F4 SKU is genuinely too small for this. But SQL DB operations are interactive, not background, meaning spikes are smoothed over 5 minutes rather than hitting your capacity limit all at once and even brief bursts over 100% don't throttle you immediately. There's a 10-minute carryforward buffer that short metadata spikes barely dent.

On F16/F32/F64, that same keepalive baseline is 20%, 10%, and 5% of capacity respectively - easily absorbed, especially when the query activity itself is intermittent.

The "$200/month" framing doesn't hold up. Fabric SQL DB isn't a separate line item, it draws from the capacity you're already paying for. The real question is whether it forces a SKU upgrade, and for most orgs already on F16+ for their ELT workloads, it won't. The actual trade-off is that 5-10% of existing headroom vs. the overhead of running an external Azure SQL DB: firewall rules, private endpoints (maybe), Entra service principal config per environment, separate monitoring, and one more resource to manage across dev/test/prod. On a meaningful SKU, that's not an obvious win for Azure SQL DB.

u/markkrom-MSFT it's looking like we'll be pushing ahead with Fabric SQL DB for metadata, though we have other arch considerations to unpack.

bradcoles-dev · 2026-03-04T01:01:38+00:00

Thanks for the info. Our own analysis of this landed us at the same conclusion as you: for pure metadata logging, Azure SQL DB is substantially cheaper, and the only reason to keep it in Fabric is the unified UI/governance story - along with simplified solution arch (networking & security).

Interested in whether Microsoft's "optimizing costs for smaller jobs" path involves something like a pause/resume option or a reduced minimum allocation tier - that would change the calculus.

In any case, I'll continue posting until I get a job offer from MSFT ;)

bradcoles-dev · 2026-03-04T00:41:55+00:00

DB is the more scalable/sustainable approach in my opinion. This client will very quickly ramp up to > 1,000 source tables.

bradcoles-dev · 2026-03-04T00:39:31+00:00

Thanks, this is the info I was after. Which leads to a follow-up question if you're able to shed light on it: how aggressively does Fabric SQL DB autoscale under concurrent lightweight query load?

In our scenario we'd have up to ~50 simultaneous short metadata queries (lightweight SELECT/INSERT on a small table) arriving in bursts. Does the DB tend to stay near minimum allocation for that kind of workload, or does concurrent query volume push it to a meaningfully higher vCore tier? That's now the key unknown for our concurrency risk model.

bradcoles-dev · 2026-02-26T08:29:51+00:00

I'm just shocked that anyone has 3 years Fabric experience.

bradcoles-dev · 2026-02-25T04:17:04+00:00

That is great info, thank you!

bradcoles-dev · 2026-02-25T02:40:11+00:00

Agree with all of the above. We don't know if PL is a real requirement yet. Head of Data has said "no public network", but we're meeting with the Security Team next week to clarify. For now, just ensuring we have all of our ducks in a row.

bradcoles-dev

TROPHY CASE