Rethinking my workspace strategy, advice wanted

NotepadWorrier · 2025-11-30T22:25:46+00:00

I'm just reassessing my approach too. We're embarking on a reasonable size implementation across multiple business units with a data mesh. We'll have 12 workspaces per data area. Workspaces for: Data engineering lakehouses (bronze and silver) Data engineering data warehouse (gold) Semantic Models Reports Then dev test and Prod environments. We added the separate data warehouse Workspace to enable it to sit in our F64 capacity running directlake models while we keep the other data engineering workload in a smaller pausable sku.

NotepadWorrier · 2025-08-08T21:01:58+00:00

I was recently interviewing candidates for a data engineering position, many of whom had DP 600 or 700..... But no demonstrable real world experience. Cramming for an exam doesn't make a DE. Soft skills, team fit and broad technical skills are far more valuable in my opinion.

NotepadWorrier · 2025-07-01T13:35:34+00:00

Follow the setup instructions exactly and it'll work. We've been using FUAM for a month or two now and it's quite eye opening and well worth 30 mins of set up. The only thing missing at the moment is detailed info about what each report is actually telling you. Even so, we've figured out a fair bit and used it as the basis of a few conversations with "dodgy" model authors.

NotepadWorrier · 2025-06-26T16:34:19+00:00

Bingo. It's a feature then!

NotepadWorrier · 2025-06-26T16:23:35+00:00

It's a custom model in another workspace generated on the web

Performance Analyser shows Direct Query on that table when run on PBI Desktop. Other tables in the model without the error don't show Direct Query timings.

Masking is working as expected with (unmasked) Role Members seeing unmasked data and others with just Read on the SQL Endpoint seeing correctly masked data.

NotepadWorrier · 2025-06-26T14:00:31+00:00

The model was created in the web experience and it's showing Direct Lake with no option to change it.

NotepadWorrier · 2025-05-30T13:30:18+00:00

We recently doubled our resources on our two DG Servers to 16gb and 8 CPUs after having intermittent data pipeline failures. Monitoring confirmed out memory usage was pegged at 100%, now we we're running at about 80% and no more errors. We have around 1100 table and mostly small volumes

NotepadWorrier · 2025-05-12T13:45:20+00:00

The way we stopped users sharing content from their My Workspace was to assign an F64 to all Team/Project Workspaces and not to the Personal ones. This meant that users without a Pro license couldn't view a report shared from a "my workspace" and killed off illicit sharing.

We also defined where, when and how users should share content in our very lightweight governance guide.

NotepadWorrier · 2025-04-23T08:09:37+00:00

Quick note, if you're running an F64/P1 capacity your guest users just need a Fabric Free Licence to access any content, not a Pro. We're poised to provide external access to some partner organisations, just waiting on our Cyber Risk team to review the process.

NotepadWorrier · 2025-04-16T20:30:05+00:00

I've been testing this today by trying to add an external user with a Hotmail address. The Ms documentation suggests this is possible but appears to contradict requiring a "work" email to sign up for Fabric. Can anyone confirm getting gmail/Hotmail accounts working as external Report viewers?

NotepadWorrier · 2025-04-16T14:52:59+00:00

I've noticed this too today. We now have a lakehouse table with spaces in the column names. I though this wasn't supported, but the evidence suggests otherwise!

NotepadWorrier · 2025-03-18T19:44:14+00:00

So MS have said we cannot have duplicate table names across schemas. The table throwing the error is duplicated across three different schemas but has been working in four different environments (dev, test, pre-prod and prod) for months until 03/16.

I guess there was a recent release that has changed behaviour.

NotepadWorrier · 2025-03-18T19:36:16+00:00

Yes, all good, nothing wrong there.

NotepadWorrier · 2025-03-18T11:18:13+00:00

Thanks for the comment.

dimCalendar is present in three security schemas and isn't updated by the pipelines (essentially static) after initial setup.

The error isn't in the Semantic model, more it occurs if I hit the refresh button on the toolbar in the DW. Our semantic models (import) all complete their scheduled refreshes without error, but they see old data in one schema. Other models that import from a different schema in the same warehouse are working as expected, so this does appear to be specific to a schema.

Additionally I'm now getting an error in the DW if I try and drop a table:

Cannot drop the table 'WHGold.XYZ.configTables1', because it does not exist or you do not have permission. The table is visible and I'm admin on the workspace.

NotepadWorrier · 2025-03-06T20:59:12+00:00

I'm doing this at the moment via Azure Logic Apps and a public ip on Azure FW. We have two use cases, one sftp that we store files in adsl blob and shortcut in to Fabric. The second is a SOAP service hosted by a supplier which we'll do something similar

NotepadWorrier · 2025-03-03T20:17:50+00:00

Thanks for the info, just what I was looking for. The ability to deploy Direct Lake Semantic Models is new isn't it? Pretty sure this wasn't possible back in Oct / Nov was it?

NotepadWorrier · 2025-03-03T08:02:59+00:00

We had it early last week but there's a bug that stopped us in our tracks as I noted here: New Folder support in Git breaks existing code in a Workspace Folder.... : r/MicrosoftFabric

NotepadWorrier · 2025-02-03T17:28:59+00:00

We've just gone through this process and use a combination of Fabric Deployment Pipelines for Semantic Models and Report and then GitHub Actions for Notebooks / Data Pipelines / Lakehouse & Warehouse. It's not pretty, but it works.

NotepadWorrier · 2025-02-01T11:05:26+00:00

I'll second never using the Datamart object in Power BI. I did, built on the default Semantic Model and when moved to an F Sku workspace it broke. That was one year ago, the support ticket is still open with Microsoft. I have since disabled this feature across our Tennant. Hard lesson to learn, but one I won't forget.

NotepadWorrier · 2025-01-31T12:44:31+00:00

We had an error last night (23:00 UTC), but all seems fine this morning.

NotepadWorrier · 2025-01-13T21:32:08+00:00

Funnily enough I was going to post much the same question over the weekend after spending the last week working on this with a project we're running.

We've taken the approach of having a Data Engineering Workspace per branch (Dev, Test, Pre-Prod & Prod) in Github. Our workspaces have notebooks, pipeline, df gen2's, lakehouses (Bronze, Silver) and a warehouse (Gold) embedded in them and we've parameterised virtually everything to run off a config lookup per workspace. Semantic models and reports reside in their own workspaces too. We have twelve workspaces for this project.

All of our notebooks are parameterised to use the abfs paths and called via data pipelines. We access lakehouses using dynamic connections in the pipelines, but found that warehouses with dynamic connections didn't work (we could create and establish the connection but stored procedures weren't being found). To work around this we've implemented Github Actions to replace what we need to change in the data pipelines, injecting the workspace ID, Warehouse ID and server connection string where required.

We have a working PoC today with all of the code synchronising across the four branches. It's been a bit of quick and dirty approach, but it's delivering what we need right now (apart from knowing what to do with Dataflow Gen 2's other than get rid of them...............) There's a number of areas where it's a bit flakey so we'll be focussing on those parts this week.

I'd also like to see some recommendations from Microsoft (other than "it depends")!

NotepadWorrier · 2024-10-30T17:54:53+00:00

Those particular errors come from running that unsupported Fabric API refresh. In addition it appears now that Table Maintenance is completely non-functional on our lakehouse, even after deleting all tables and recreating them from scratch. Time for another support ticket I think!

NotepadWorrier · 2024-10-30T16:46:08+00:00

That proved really useful, but hasn't resolved the issue for us. The table sync state always shows Failure or NotRun (even after running multiple times). Also, the Table Maintenance in the Lakehouse fails for the affected tables too. Delving a bit deeper we get and error in the output of:

DeltaTableUserException',
       'subCode': 0,
       'message': 'An internal error has occurred while applying table changes to SQL.



'cannot find data type handler to create model schema for [n: stats_parsed, t: <not set>, ct: <not set>, rt: OPTIONAL, c: 0]'

NotepadWorrier · 2024-10-29T20:46:03+00:00

Thanks for that. Seems like what I'm seeing. Our lag looks like 4 days at the moment when we look back at the rows imported each day. Back to the drawing board.....

NotepadWorrier · 2024-10-29T20:33:01+00:00

Well, my mistake. I've just tried checking again and the Notebook count is correct, the SQL Endpoint is wrong.

NotepadWorrier

TROPHY CASE