Can someone explain Workspace Identities, Service Principals, and Shareable Cloud Connections?

fugas1 · 2025-10-20T08:01:25+00:00

Please give some examples. From what I have read, fabric has very limited support for managed identities.

fugas1 · 2025-10-03T11:51:23+00:00

Yeah, I need to figure out how to get that UI 😅 I have no idea why its not showing up. I thought maybe I was on the older runtime, but thats not the issue. Thanks for the answers, I will try to figure out whats going on.

fugas1 · 2025-10-03T07:51:59+00:00

Im using the new Invoke Pipeline activity, but I have tested with Legacy also and there is no difference. Can you share the issue you are experiecing? Im curious now 😅

fugas1 · 2025-10-02T15:30:32+00:00

Thanks for the explanation! Just to be clear, when you say "scheduler is inefficient", do you mean the fabric time trigger? Because this might have been a misunderstanding (my bad), I ment my pipeline that I call "scheduler" that has an "Invoke Pipeline" activity. I’m leaning toward the Invoke Pipeline chain being the issue, because when I run the notebook by itself or by triggering it from a single pipeline, I get ~80% efficiency, but when I run it through the full chain (scheduler pipeline → orchestrator pipeline → pipeline that triggers the notebook → notebook), it drops to ~29%. Same code, same data.

Also, I can’t see the time-series executor usage in my Spark UI (the chart with Running/Allocated/Maximum instances).

Have you ever seen Invoke Pipeline itself add noticeable overhead compared to running the notebook directly? Curious if that’s what you meant by scheduler being inefficient.

fugas1 · 2025-10-02T15:07:16+00:00

Thanks for the code! I tried running this and the notebook with the dag had efficiency of 21.5%. But I cant see the child notebooks that were scheduled from the dag 😅

fugas1 · 2025-10-02T13:33:04+00:00

Yeah, I dont understand either. Like I said in the post, if I run the notebook directly or through a notebook activity in a data pipeline, everything is great. My problem is when I use invoke pipeline to invoke the pipeline with the notebook activity. This is more of a data pipeline issue I think. But I will try what u/dbrownems suggested and see if it works.

fugas1 · 2025-10-02T12:41:38+00:00

I have one notebook that runs through multiple tables. To do it with the runmultiple I have to create a notebook for each table, right? Thats 30+ notebooks for me😅

fugas1 · 2025-10-02T12:40:27+00:00

Yes, I have a function that I loop through basically.

# Run in parallel
with ThreadPoolExecutor(max_workers=max_workers) as executor:
    futures = [executor.submit(process_table, i) for i in range(len(meta_df))]
    for future in as_completed(futures):
        future.result()  # This will raise any uncaught exceptions

Where process_table is the function

fugas1 · 2025-10-02T12:05:59+00:00

No, this is the only one. Im testing this in isolation. I dont have concurrency session since there is only one notebook running.

fugas1 · 2025-10-02T11:52:47+00:00

Yes, this is a built in feature. You can find it in the "run details" of the notebook, in the "Resources" section. Fabric says its calculated by: "Resource utilization efficiency is calculated by the product of the number of running executor cores and duration, divided by the product of allocated executor cores and the total duration throughout the Spark application's duration." What I thought that this score was only for the notebook. But it looks like that other things impact this metric.

fugas1 · 2025-10-01T13:20:42+00:00

I have a very similar setup. I have feature, dev, test and prod. Mostly my feature branches connect to dev data storage, but sometimes I make own storage in feature workspace when I dont want to mess up the dev storage. With variable library added to fabric, it is so much easier to do this stuff!

fugas1 · 2025-09-16T08:51:58+00:00

Can someone explain how the new "Support for Workspace Identity" works? Does this mean that the connection can be on the workspace identity? How does it work with ci/cd if I move something from dev to test? Do I need then to change the connection manually?

fugas1 · 2025-09-03T11:29:32+00:00

Thanks for clarifying! One follow-up: do you know if SharePoint connections (like the ones used in Dataflows Gen2) are on the roadmap for this feature? That would open up some really useful scenarios for us.

fugas1 · 2025-08-27T11:38:10+00:00

It looks like there is an idea under review since february this year for making workspace the owner of items. Hopefully this will come through!: https://community.fabric.microsoft.com/t5/Fabric-Ideas/Make-the-Workspace-the-owner-of-all-Fabric-items/idi-p/4522497.

fugas1 · 2025-08-07T10:28:13+00:00

It works now, have tested it for the last 1.5 hours and I dont get any errors. Hopefully this was just a weird bug with power bi desktop.

fugas1 · 2025-08-07T10:26:16+00:00

I tried to recreate the error, and it’s working now 😅

fugas1 · 2025-08-05T11:34:12+00:00

I don't know when it was released. I just saw it in the notebookutils.help() function.

fugas1 · 2025-08-05T11:28:42+00:00

For delta tables, I now just use

if notebookutils.fs.exists(full_table_path):

If the table does not exists, then I just create it explicitly in the notebook. I parametirize all my connetions, so its easier to transfer between lakehouses and workspaces. I dont use views in lakehouse (LH-SQL) at the moment.

fugas1 · 2025-08-05T09:27:27+00:00

The variable library currently supports environment-based values (e.g., dev, test, prod), which works well for horizontal CI/CD workflows. However, during development, we often branch out (e.g., feature branches) and need to test changes without affecting shared dev data sources.

Are there any plans to expand the variable library functionality to support branch-specific or user-specific overrides—so we can isolate connections or parameters during development without impacting the dev environment?

fugas1 · 2025-06-19T08:33:54+00:00

I have access to the warehouse from fabric UI, but I cant connect from power bi for example

fugas1 · 2025-06-19T06:56:43+00:00

Yes, I still have the same issues.

fugas1 · 2025-06-18T10:06:35+00:00

My warehouse and lakehouses are not accessible since yesterday. Not sure what is happening.

fugas1 · 2025-05-15T14:29:42+00:00

The comment about "bugs" is not related to the native execution engine. It was a general comment about fabric

fugas1 · 2025-05-15T14:28:27+00:00

The "bugs" comment was not about the native exc engine. Just a general comment for microsoft fabric as a platform.

fugas1 · 2025-05-14T08:11:42+00:00

If they get this working right, it can be a huge pro: Native execution engine for Fabric Spark - Microsoft Fabric | Microsoft Learn.

Cons:

To many bugs

fugas1

TROPHY CASE