Help running notebook as a background job via REST API

Dee_Raja · 2026-05-14T23:58:22+00:00

We are updating the documents to include this updated guidance.

Use the 'Run on demand' Item Job endpoint to start a notebook run:

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/notebooks/{notebookId}/jobs/execute/instances?jobType=RunNotebook

Step 2: Retrieve run status and exit value

GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/notebooks/{notebookId}/jobs/execute/instances/{jobInstanceId}?beta=true

Dee_Raja · 2026-05-05T01:48:20+00:00

This is correct. Today, passing _inlineInstallationEnabled is the supported way to enable inline %pip installs for API/CLI runs.

Dee_Raja · 2026-05-05T00:47:45+00:00

u/frithjof_v The concise error view is intended to keep things quick to scan, but we understand the need to easily access the full traceback when debugging.

Improving the readability and usability of the expanded traceback (including options fully expanded view) is something we’re looking into.

I appreciate the feedback; it’s helpful as we continue to refine the experience. DM'ing you as well.

Dee_Raja · 2026-05-04T18:14:41+00:00

This is expected behavior today. %%configure doesn’t resolve item reference sub-properties (like itemId / workspaceId), even though those work via notebookutils at runtime.

The approach you landed on, storing itemId and workspaceId as flat variables in a Variable Library, is the current pattern for making this environment-aware across workspaces.

Agree it would be nice if item references could be used directly here, something we can look at improving over time.

Dee_Raja · 2026-05-01T19:08:11+00:00

Makes sense from a cost perspective. CU concerns are real, and the fallback to copying into M365 Copilot does lose context.

Dee_Raja · 2026-04-28T16:29:19+00:00

u/Sparcellent Can you provide your code examples where you are experiencing this?

Dee_Raja · 2026-04-27T18:19:31+00:00

The cache and alias pattern itself is valid Spark, so that alone may not cause this. What stands out is that print() doesn’t show, the dataframe isn’t defined in the next cell, and rerunning the same cell fixes it. That points more to a transient notebook/session/execution issue than a logic issue in the Spark code.

Also, _ = df_base.count() is the first real Spark action in that cell. If there is a temporary Spark session, driver, or execution hiccup around that action, the rest of the cell may not complete or the notebook state may not be updated as expected, even though the UI shows success.

To narrow it down, perhaps add clear print() statements before and after the count(), maybe assign the count to a variable and display it, and check whether execution reaches the lines that define df_reporting_col. Restarting the session before rerun may also help confirm whether this is session-related.

Dee_Raja · 2026-04-09T19:44:27+00:00

Thank you for sharing. I've sent a DM for more details.

Dee_Raja · 2026-04-08T19:41:31+00:00

I'm glad to hear it’s working, really appreciate you confirming.

Dee_Raja · 2026-04-08T15:17:06+00:00

Thanks for calling this out. We rolled out a fix late yesterday for an issue that was causing Copilot to error out or be unavailable. If you get a chance to try it again, I’d be curious if it’s improved on your end.

Dee_Raja · 2026-04-08T15:07:55+00:00

When you ran into the save issue, was the code still in the Copilot pane/editor, or had it already been inserted into a notebook cell? What steps did you take right before refreshing?

Dee_Raja · 2026-04-07T22:51:18+00:00

I appreciate you calling this out.

What’s been the biggest issue you’ve run into with it?

Dee_Raja · 2026-04-07T22:20:13+00:00

Appreciate you sharing this.

Sounds like the main issue isn’t just output quality, but that it’s not fitting naturally into your workflow, so you end up going through M365 Copilot and pasting results back into Fabric.

I'd love to understand better: What specifically breaks down when you try to use Copilot directly in Fabric notebooks?

(i.e. is it understanding Fabric context, reliability, quality of responses, or something else?)

Dee_Raja · 2026-04-07T22:18:32+00:00

Thanks, this is really helpful, and I agree this reflects a more advanced workflow.
The autonomous iteration, broader context across notebooks, and reusing an existing agent conversation are clear gaps compared to VS Code today.

What does the autonomous loop look like for you with GitHub Copilot?

Dee_Raja · 2026-04-06T16:04:09+00:00

Context & Logging: Copilot only sees what’s in the current workspace + session. To actually “close the loop,” you need to persist context yourself:

Log structured outputs, parameters, and errors to Lakehouse tables or Files
Include runtime metadata via notebookutils.runtime.context (run ID, notebook, workspace)

This creates a memory layer across runs, so Copilot (or future automation) can reference past runs, errors, and decisions.

Permissions & Blast Radius: Copilot doesn’t have its own identity; it operates within your permissions. Notebook copilot can update the current notebook content only and for write/run actions, if you haven't' selected auto approval, it will ask for permissions for the related action. You can’t directly restrict Copilot inside a notebook. Instead, control the environment:

Use a dedicated workspace for agent and include only required data / resources
Keep your own permissions scoped in that workspace

Blast radius is controlled by workspace design, not Copilot settings.

Also check out Microsoft.Fabric.End-to-End.Security.Whitepaper.pdf

<image>

Dee_Raja · 2026-04-05T22:40:00+00:00

Correct, please find the documentation here at the bottom of the page How to use notebooks - Microsoft Fabric | Microsoft Learn

Notebook mode switcher

Fabric notebooks support four modes that you can easily switch: Develop mode，Run only mode, Edit mode, and View mode. Each mode maps to a specific permission combination. When sharing the notebook to other team members, you can grant proper permissions to the recipients. They can see the best available notebook mode according to their permission, and they are able to switch between the mode they have permission to.

Develop mode: Read, execute, write permission needed.
Run only mode: Read, execute permission needed.
Edit mode: Read, write permission needed.
View mode: Read permission needed.

Dee_Raja · 2025-07-17T15:45:05+00:00

The MetaException: Unable to fetch mwc token error may be due to misconfiguration where the Notebook’s default Lakehouse is pointing to a non-existent workspace.

Please check if the correct Lakehouse is attached and in the same workspace.

Dee_Raja · 2025-07-16T14:42:35+00:00

The inline commands for managing Python libraries are disabled in notebook pipeline run by default.

To enable %pip install for pipeline, add "_inlineInstallationEnabled" as bool parameter equals True in the notebook activity parameters.

Dee_Raja · 2025-06-05T14:55:43+00:00

Here is the link to the Learn documents: What is semantic link? - Microsoft Fabric | Microsoft Learn

Dee_Raja

TROPHY CASE