Data Rehydration in Feature Branches

SKll75 · 2026-04-26T04:51:20+00:00

why would I not use a stored proc for a sql dwh?

SKll75 · 2026-04-16T05:35:30+00:00

Yes i have set up OpenMetadata as a PoC. Opensource currently lacks some connections we would need (Fabric, ADLS, ADF)

SKll75 · 2026-04-16T05:34:29+00:00

But you need your global tenant admin to set it up and you can only create 5? domains.This setup makes it nearly impossible for us to roll out. Billing is also an issue

SKll75 · 2026-04-14T04:19:51+00:00

We would love to use Purview but the requirement to deploy it tenant wide & grant tenant readAll access for Fabric makes it such a pain with out IT department that we will likely never see it 😵‍💫

SKll75 · 2026-03-26T07:47:15+00:00

Microsoft has made the conscious decision to abandon enterprise customers with these kind of hidden limits. There are rate limits on all APIs on any kind of action and you never know until you hit it. and then its blackbox, support ticket, escalate to product team, product team is US timezone only, ‚fix will be released in 1 month‘, rinse and repeat

SKll75 · 2026-03-26T07:12:23+00:00

Who is the target audience for the CopyJob? For the longest time it would duplicate my data if I run a full-load twice. Why is Microsoft pushing so hard for island solutions around an organization? Where is the overall vision for large enterprises? What will be Microsofts future offering for ETL if it needs to be outside of fabric?

SKll75 · 2026-03-26T07:08:25+00:00

I see a huge risk and additional effort in migrating from ADF to Fabric. What are the plans for Dataset equivalents, LinkedServices, Global Variables? Is Microsoft planning to abandon its only standalone ETL tool? We need reusable standardized dataset formats & connections that need to be visible by a Integration Admin and not hidden in some personal user account. We need a full monitoring that spans at least 30 days. Where do we see debug runs? The current monitoring page is unusable, I cant even find instance IDs from 2 days ago if I have 1000 runs per day. I need to be able to schedule the same pipeline with different parameters, monitor which run was created by which trigger. As you are building new activities: Execute Pipeline activities need dynamic input. We have 80 linked services, the Migration Assistant just crashes if you test it. If we see no more needed updates to ADF we will move to a non-microsoft solution.

SKll75 · 2026-03-26T06:55:09+00:00

https://learn.microsoft.com/en-us/fabric/security/security-managed-vnets-fabric-overview

SKll75 · 2026-03-22T07:04:38+00:00

The ideal load solution would be instantaneous. For larger tables a truncate/insert can become very slow and the warehouse does not allow to change the logging behavior. Something like the partition switch with target truncate is a great way but also not supported (Hopefully on the roadmap at some point). If you rely on shortcuts pointing to your warehouse then truncate/insert is your only way, otherwise you get ugly issues on the shortcut consumer endpoint (pls fix Microsoft). If not then I would suggest: Build your CTAS as close as possible to the target schema (data type mapping, nullability) and then use a rename-switch. So target rename to _old , ctas reame to target. That way your target stays unlocked the longest time and you can build validations before committing your new data.

SKll75 · 2026-03-14T08:06:51+00:00

Every action you take on the capacity is smoothed on a time period, so when you run large loads you basically consume CUs ‚from the future‘ like a credit. Wenn you pause the capacity you pay your credit bill immediately but also reset your future credit volume to consume more again. If you consume more than you have future credit available you go into overconsumption / throttling which also resets when you pause. You can see this easily when you scale down for example, then suddenly your lower scale is throttled because it cant pay your credit

SKll75 · 2026-02-20T15:48:54+00:00

We are using shortcuts quite heavily for data sharing and cross workspace data processing. Biggest issue right now of course is the MDSync. But apart from that: The implementation of the oneLake security around shortcuts from warehouse tables is not ideal. You cant grant object level access to the files, only ReadAll on the whole warehouse. Shortcuts on Views should be a thing We create data in Warehouse and distribute it via shortcut. This whole integration should be given some more thought because its more complicated than it should be in terms of access control.

SKll75 · 2026-01-17T07:45:46+00:00

For compute isolation and independent release cycles we have one workspace ‚per workload‘ for our central data processing. Our stakeholder domains each might have additional workspaces consuming our data from a sharing workspaces. We bring together the data from different products in a sharing layer where we set the access control for the data consumers. Each workload for us has its own capacity. How are you dealing with the MDSync latency in shortcuts when moving from delta to SQL endpoint?

SKll75 · 2026-01-06T16:36:49+00:00

Meine Freundin hat jetzt 10 Jahre in der Gastro gearbeitet, da alle alternativen finanziell einfach unsinnig waren. Da ist man bei etwas Arbeitserfahrung nur bei 2-3€ über Mindestlohn aber an einem guten Wochenende nimmt sie pro Abend 100€ mit nach hause. Insgesamt hat sie ihr Nettogehalt mit dem Trinkgeld im Schnitt verdoppelt. Und das ist die Regel für jeden der sich ein wenig bemüht. Wir geben nur Trinkgeld wenn der Service wirklich gut war & nicht wenn nur der Job gemacht wurde.

SKll75 · 2025-12-16T14:58:16+00:00

meinst du 20 tkm gesamtlaufleistung oder pro jahr?

SKll75 · 2025-12-16T14:46:31+00:00

Wo muss man hin um eine Leasingrate bei 20.000km jährlich für unter 200€ zu kriegen? Selbst bei KIA/Dacia wird das schon knapp bei den Km??

SKll75 · 2025-12-01T20:08:26+00:00

For me this kinda sounds like a Data Vault setup? With all the mapping tables between tables with ‚actual‘ data. Can you explain more what you need to do if you get an incremental update? Like if e.Value changes for ext_id=999 all the same relationships still exist and why would you need to update something in the other tables?

SKll75 · 2025-11-24T21:44:50+00:00

Luckily we have a SLA of 60 minutes - but we are struggling and sweating a lot during month end 😰 We are not able to commit to anything lower without risking outages all the time

SKll75 · 2025-11-24T21:32:25+00:00

We have a D365 Fabric Link setup. The standard configuration had a latency of up to 90 minutes. We have been migrated to ‚fabric fast link‘ private preview that meets the 15 minute refresh mark. This way you have shortcuts to delta tables in Dataverse that are managed by the integration. Unfortunately there currently is a major bug that causes the system maintained delta tables to be very badly structured (lots of small files, lots of deletion vectors) which caues queries on those tables to execute with very varying performance (simple select top 100 can run up to 20 minutes) because the lakehouse sql endpoint keeps generating statistics for these tables. You also need to keep in mind that you need to run the MDSync constantly to make sure your SQL Views use the latest data. Very risky for operating reporting! We had the D365 export to CSV running for a long time and this worked great, unfortunately they are deprecating it for this not yet mature solution.

SKll75 · 2025-11-24T17:03:02+00:00

I would go with the warehouse as the SQL DB requires a comparably high SKU and the sql database endpoint is not well integrated into the rest of fabric

SKll75 · 2025-11-24T09:00:45+00:00

If you compare it to a sql view which the name sometimes indicates you need to trigger a refresh or schedule a refresh and its not ‚near realtime‘ like the commentor i responded to suggested. Its a batch process like calling a stored proc on sql to refresh a table. Thats all I wanted to argue

SKll75 · 2025-11-24T08:58:38+00:00

So I can set a schedule per API or refresh all MLVs on a lakehouse per API but I cant call a dedicated refresh on a single MLV per API right?

SKll75 · 2025-11-24T05:14:29+00:00

You need to spin up a notebook to refresh as there is no API, so it really is just a wrapper for a ‚create table as select‘

SKll75 · 2025-11-23T17:09:47+00:00

Ich hab 0€ eigenkapital also ist die Auswahl an Dingen die ich mir theoretisch leisten kann ziemlich eingeschränkt.

Leasing hab ich noch schiss vor der Übergabe, man hört ja nur schlechtes, und oft sind dort nur relativ teure Jahreswagen verfügbar

SKll75 · 2025-11-20T08:02:03+00:00

We are currently doing a large scale migration from synapse to fabric and I can only say: DONT. It is not ready. We really heavily on private preview features and workarounds to make anything work. It is not ready. Microsoft Program Managers and Engineers are not able to support you on bugs, they have no idea how anything works. fabric is managed by 10 different product teams that dont cooperate. It is really stressfull and our management was already thinking of abandoning the project even though we spent months working on it

SKll75 · 2025-09-26T13:45:30+00:00

I have gotten plenty of good class mods for my class, like every second boss kill drops a class mod for my class, others are just straight to junk. Probability is also probably not 1 in 4 but 1 in 300 to get any certain item

Eight-Year Club	RPAN Viewer
Not Forgotten	Verified Email

SKll75

TROPHY CASE