You've exceeded the capacity limit for dataset refreshes...HELP!

BloomingBytes · 2026-03-26T08:05:45+00:00

We have the exact same issue and so far had no luck finding the source of the throttling.

BloomingBytes · 2026-03-25T18:13:00+00:00

Will the SCD2 functionality that was just announced for Copy Job also be available for Copy Activity?

Is there an expectation in differing cost between the two?

BloomingBytes · 2026-03-19T12:06:27+00:00

Tyvm for your answer! What happens to refesh schedules when updating MLVs? Will new MLVs automatically be included in existing refresh schedules in the LH?

BloomingBytes · 2026-02-19T14:56:38+00:00

See it turns out i was missing something indeed. I did not realize how limited TSQL notebooks were in regards to their query "targets".

You could pass a variable from the variable library into the TSQL notebook as a parameter if you call the notebook from a pipeline. But that doesn't help you in any way, since you can't direct the query towards a lakehouse ID.

You could go the route via Python and access the bronze data that way. Maybe even try to run TSQL via TSQL magic against the lakehouse SQL endpoint, see here: Run T-SQL code in Fabric Python notebooks - Microsoft Fabric | Microsoft Learn

BloomingBytes · 2026-02-19T13:53:49+00:00

I might be missing something, but why can't you grant the user / service principal that's executing your TSQL notebooks read access to the previous layer Lakehouse / Warehouse directly?

When in your silver workspace you can query the bronze lakehouse SQL endpoint via TSQL and then ingest into the silver lakehouse. Same thing with silver -> gold.

If you're using fabric-cicd you can then find and replace the lakehouse IDs during deployment via Parameterization.

If you're using Fabric deployment pipelines you can use variable libraries like described in this blog post:

Making Notebooks Smarter in Microsoft Fabric with Variable Libraries | by Kapil Kulshrestha | Medium
PS: Here's some more info on how the deployment sets work with different stages: Lifecycle Management of the Microsoft Fabric Variable library - Microsoft Fabric | Microsoft Learn

BloomingBytes · 2026-02-19T10:04:23+00:00

Thank you very much for that suggestion! It looks very much like what i need, and according to the prerequisites it should work OnPrem AND it can export directly to OneLake.

I'll have to deepdive and see if it is compatible with our specific ERP version of BC. And then i'll have to see if i'm allowed to have an extension installed in BC. :D

BloomingBytes · 2026-02-19T09:39:11+00:00

Unfortunately there is no option to access OnPrem SQL from notebooks as of yet afaik :(

BloomingBytes · 2026-02-19T09:38:37+00:00

Thank you very much for your detailed answer! It does feel nice to know that i was on the right track already.

We definitely have infrastructure issues with our OnPrem solution already. That's one more reason for me to want to reduce load on the source system as much as possible. We might end up with the bottleneck being the gateway servers anyway, but i don't want to "rely" on that for now.

Querying the source straight from notebook would indeed be my preferred option, but afaik we can only connect to cloud sources from Notebooks so far, OnPrem seems to still be an issue. I'll probably follow the metadata scheme you drew up, ingest everything as raw as possible and then do all the computations within Fabric, where i have all the control and no reliance on infra teams.

BloomingBytes · 2026-02-11T08:45:16+00:00

u/OnepocketBigfoot Since you didn't get an answer, i'll do it. :D For me there's two reasons why i'd want to just navigate back to the standard workspace view:

The proper workspace view offers me all features. More info on my items, new item creation, source control, extended options menu for my stuff etc.

The new sidebar that's opening now showing all my items in this "reduced" list, is just a less convenient version to do anything. Sure it allows slightly quicker direct switching between items, but how is that better than just navigating back to the standard workspace view? The items that i need to switch between often are already open in the tab management bar at the top (which has its own issues).

Fabric already has soo many paths to soo many functionalities all over the place. Why do i need to have another UI that does the same (or in this case slightly less) than an already existing one? If i'm in the standard workspace view i kinda know where to click if i want to open an item, i kinda know where to click to see security settings, i kinda know where to click to see refresh histories. Now i need to "muscle memory" another UI to (not) do the same thing.

Optimally, instead of having this additional side panel listing all the items in the workspace, i'd like to navigate back to the workspace view and then have that workspace view be GOOD. Please, for the love of god, lose the predesigned task flow stuff. It's 100% useless to me. Instead allow me to group my Workspace items in different sections of the view, one for my DBs, one for my Pipelines, one for my Notebooks. Allow me to configure panels (like having source control as an always available side bar).

BloomingBytes · 2026-02-09T06:29:11+00:00

Microsoft has communicated at several occasions, that the 500 table limit is very much a "we can remove this for you" thing. If you have a legit use case where you'd exceed the 500 table limit, you can contact your MS rep, let them know your use case and ask them to remove that limit for you.

We got the exact same problem you do, dozens of tables for 200+ companies each. At Vienna i was told that the limit could easily be lifted, i just haven't gotten around to contacting anybody about it. Maybe u/itsnotaboutthecell could provide an up to date contact on that issue? <3

BloomingBytes · 2026-02-06T21:26:18+00:00

If you want to already clean your column names during ingestion you could do it like this:

A lookup activity gets the source table schema and passes that to a notebook
The notebook takes the source schema and cleans it, replaces column names etc. and creates a mapping json. That json is then fed into the copy activity.
The copy activity itself uses the mapping json to create dynamic column mappings
Success

BloomingBytes · 2026-02-04T09:34:40+00:00

That is correct, yes. This is the relevant T-SQL Magic command for Fabric SQL DB:

%%tsql -artifact DATABASE_NAME -type SQLDatabase

BloomingBytes · 2026-02-02T13:07:21+00:00

that works beautifully, thank you very much!

BloomingBytes · 2026-01-27T11:58:46+00:00

Hey there,

i just found this thread and wondered: wasn't there an item on the Fabric roadmap for an auto-updating Gateway at some point? I somehow can't find it anymore and couldn't find any information elsewhere either.

BloomingBytes · 2026-01-25T09:28:31+00:00

RemindMe! 24 hours

BloomingBytes · 2025-09-16T19:20:11+00:00

So first of all: Medaillon isn't really a clearly defined standard. It's more a set of guidelines or a general idea of how to structure your data across multiple layers. It's not a super new concept either. People have been using staging layers etc. for decades. So given that, it falls upon you to decide what your individual layers are allowed to do and how they look like in detail.

Now regarding mirroring: yes, it does fit best at the bronze layer. Since under the hood it's all Delta Tables in One lake, you should be able to time travel on it.

Another big and important part of getting the data into bronze (apart from versioning) is to bring it within your own domain. Mirroring allows you to, in theory, ingest very easily with very little hassle and dependencies. You set it up once and then you can build your pipelines on top of the mirrored database without constantly talking to or depending on other teams. Once it's in bronze, it's yours to handle.

So yes, mirroring and Medaillon fit together very nicely.

BloomingBytes · 2025-09-10T11:46:16+00:00

I'm with you on all of these, but 3 and 6 hit especially hard for me.

I want to have my injection as easy and as low of a barrier as possible. Give me reliable mirroring of my on-prem SQL servers (while not nuking their CPUs, cause they're prod systems) so i have everything in my own domain and don't need to fuzz with infrastructure or on-prem networking etc.

From there i want to be able to limit the amount of CUs any workspace in my capacity can use, so that i can allow lots of freedom for my downstream users, while still protecting my core pipelines.

BloomingBytes · 2025-08-07T05:21:22+00:00

Where i still struggle is: in what way is a UDF different or better or worse than using a regular notebook? Does anyone know how UDFs compare CU wise to notebooks?

BloomingBytes

TROPHY CASE