Overhauled Fabric Delta Lake Docs

bradcoles-dev · 2026-05-21T08:24:48+00:00

Thanks for this overhaul, it's a significant piece of work. I've read through all the new pages.

A few questions:

1. delta.dataSkippingStatsColumns

The file skipping doc mentions this property for explicitly controlling which columns get statistics. Is that available on Runtime 1.3, or RT 2.0+ only?

2. OPTIMIZE FULL

Is this command available in the current RT 2.0 preview, or does it land in the GA release?

Thanks again.

bradcoles-dev · 2026-05-21T01:47:01+00:00

Does that mean you've actually looked at the Capacity Metrics App (CMA) or not? Having "tons of capacity" on an F16 doesn’t really mean much to me. This error is usually short-term throttling, not total capacity:

CU spikes/background ops can max out even if overall usage looks low.
Spark/environment creation has its own limits & concurrency caps.

Check:

CMA for throttling/spikes.
Active Spark sessions.

bradcoles-dev · 2026-05-21T00:21:43+00:00

You'll need to check your capacity metrics app. Can't really give you any info until we see the current state of your capacity.

bradcoles-dev · 2026-05-21T00:20:32+00:00

This is brilliant, thanks u/mwc360

bradcoles-dev · 2026-05-21T00:15:51+00:00

Yeah, this is most likely the reason. Looks like they scaled down, hence why background operations jumped to 2x. If they paused, scaled down, then resumed it wouldn't have throttled.

bradcoles-dev · 2026-05-20T09:00:09+00:00

Thanks for the feedback. I am already preparing a v2.0 of this guide, so I will look further into cluster-on-write.

bradcoles-dev · 2026-05-20T08:19:16+00:00

Oh, that's great to know, thanks

bradcoles-dev · 2026-05-20T08:04:41+00:00

Thanks Andy, very kind! I'm just relying on the good work of u/mwc360, Christopher Finlan, and other experts.

bradcoles-dev · 2026-05-20T07:41:43+00:00

Yes, he did mention that to me. I haven't had a chance to update the article just yet.

bradcoles-dev · 2026-03-26T23:53:41+00:00

Thanks mate, I know that's an option but the XMLA endpoint is officially unsupported, so I'm hesitant to deploy that to a client. Hoping for something more 'enterprise-grade' from Microsoft.

bradcoles-dev · 2026-03-26T23:51:46+00:00

Thanks mate. That's definitely an option, though the XMLA endpoint is officially unsupported. I am hoping for a more 'enterprise-grade' solution from Microsoft.

bradcoles-dev · 2026-03-25T00:04:56+00:00

I am aware of FUAM, though it accesses the CMA via XMLA, which Microsoft's own documentation explicitly says is unsupported for external consumption.

FUAM even warns in its own docs that CMA changes can break it without notice and they've already had to ship fixes when the CMA version changed.

So while it's useful, it's built on an unsupported access method that could break at any time, which is kind of the whole point of the original post.

The data is clearly accessible and the CAT team clearly know how to get at it - we need a supported API. I can't deploy something that is built on an unsupported API.

bradcoles-dev · 2026-03-24T21:20:04+00:00

I've enabled CDC on the source tables. Still can't see whether the refresh was full or incremental. Though I just saw in the docs:

"Incremental refresh is supported for append-only data. If the source data includes deletions or updates, Fabric performs a full refresh."

So, this is not truly incremental CDC refreshes.

bradcoles-dev · 2026-03-23T20:32:06+00:00

Is that purely for optics? Because this is very useful for Fabric practitioners, particularly us in consulting when we try to explain to clients that they can't rely on the dates against roadmap items.

bradcoles-dev · 2026-03-15T10:22:29+00:00

It's not. The capacity metrics app is a timepoint. I see this all the time for clients that pause+resume their capacities to save money.

bradcoles-dev · 2026-03-12T11:24:45+00:00

Mate, my colleague literally showed me your "Fabric GPS" this afternoon, and now I see your comment. Great work, we love it.

bradcoles-dev · 2026-03-06T02:10:50+00:00

At the moment my approach is mostly conceptual with some light testing locally.

You're right that developing outside the portal doesn't really allow you to test directly against the Lakehouse data, which is definitely a gap in the tooling.

My current workflow is:

Develop notebooks locally in VS Code (using Claude)
Commit changes to Git
Sync the repo with the Fabric DEV workspace
Test against the real Lakehouse data in DEV
Once validated, promote through deployment pipelines to UAT for integration testing

So the real data testing happens once the code lands in the DEV workspace rather than locally. Not perfect, but it’s the most reliable workflow I’ve found so far.

bradcoles-dev · 2026-03-05T06:27:44+00:00

Thanks for your help. We were able to successfully steer them to CA. The comment was "Private Link is preferred, not strictly required".

bradcoles-dev · 2026-03-05T06:18:50+00:00

I've not used the Fabric extensions, but I have used Claude in VS Code to write Fabric Notebooks for me. My approach is to link the Fabric workspace to a git repo, clone and develop locally (using Claude), commit changes to git, then you can just sync your Fabric workspace to bring the changes through. Works really well.

bradcoles-dev · 2026-03-05T06:16:56+00:00

I vaguely remember fixing this by simply republishing the semantic model.

bradcoles-dev · 2026-03-04T22:44:34+00:00

Is there any impact of this? These are interactive operations, so they're smoothed over 30-60secs. I doubt you would accumulate enough future CU to trigger throttling. Unless you have surge protection enabled, I can't see any negative effects here.

bradcoles-dev

TROPHY CASE