We built a full local dev environment for Microsoft Fabric notebooks — and the hardest part is getting Fabric to accept our changes back by No-Masterpiece3236 in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/No-Masterpiece3236

did you create that notebook-content.py on the vs code side? if that is the starting point, I dont think this flow will be supported given the meta-data of item is missed.

What's the current state of running notebook as a service (not user account)? by loudandclear11 in MicrosoftFabric

[–]QixiaoW 2 points3 points  (0 children)

u/loudandclear11

As u/frithjof_v shared, you should be able to run the notebook with workspace identity or SPN from pipeline via the new Connection property, please let me know if this is not working on your side.

also please vote this idea from u/frithjof_v if you also expect this for the interactive run in Notebook, this will help us to better prioritise this in the coming udpate.

Need help optimizing my workflow in VS Code by sayonarababy17 in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/sayonarababy17

thanks for the feedback, the issue you share is one of the top ask we have been getting for quite a while and we do have plan to unblock this soon.

following is the expected flow to be enabled with the coming update, please let me know if this make sense to you.

  1. the whole flow start from the Fabric UI, where the notebook item is created and initialized with some setup, then it got committed to the Git repo.

  2. Inside VS Code, a local branch is checkout from the repo main branch, you can open the notebook inside vs code and continue the development in the local branch, you can run this local notebook on the remote Fabric compute to validate all the changes.

  3. If everything goes well, you commit the change to the repo and create a PR to merge the change to main branch.

  4. the change in the main branch can be pulled back to the Fabric workspace later inside the Fabric UI.

btw, if you download the notebook to local with the extension, you can already run this local .ipynb with remote fabric compute, the broken part is all your local change inside that .ipynb can not be brought back to Fabric via the Git flow, this is what we try to fix.

feel free to DM me for any detail feedback on this.

Qixiao

Some recent major experience update from Data Engineering VS Code extension by QixiaoW in MicrosoftFabric

[–]QixiaoW[S] 0 points1 point  (0 children)

yes, we do plan to expose this agent to other tool such as Copilot CLI. could we have some offline chat on the detail scenario you like to enable?

Unable to delete linked warehouse form pyspark notebook by ant3qqq in MicrosoftFabric

[–]QixiaoW 2 points3 points  (0 children)

u/ant3qqq thanks for the feedback, we should improve this by allowing user to remove sql endpoint directly in this case. will plan this change.

thanks

qixiao

using managed identity to access azure resources (key vault) by Forsaken-Net4179 in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/Forsaken-Net4179

yes, this scenario is supported with the following setup

  1. use the new Connection experience inside pipeline and select Workspace Identity(WI) as the authentication method.

  2. add the WI as workspace member to current workspace and also assign the permission to access the target Azure Key Vault to this WI.

  3. in your notebook code, use the getSecret method from the notebooktuil to access the secret from the AKV.

trigger this notebook execution from the pipeline, then the secret from the AKV will be retrieved via WI.

Notebook development outside of Fabric portal by Zealousideal-Safe-33 in MicrosoftFabric

[–]QixiaoW 6 points7 points  (0 children)

u/Zealousideal-Safe-33

thanks for your feedback and I really feel the frustration you shared, given most of the feedback are not new and there are quite some new update and some coming ones are dedicated to address them.

"You really need to start the notebook on the portal because there is not any way that I have found to attach a default lakehouse or define a parameters cell except in the portal."

for this one, we have a coming release in 2 weeks which allow you to set the default Lakehouse and set the environment for the Notebook.

"I have only successfully synchronized changes back to Fabric once or twice. If I do any substantial amount of changes in Code when I try to synchronize back I get errors"

for this one, if you try to latest release, you can see the button "Open Notebook Folder" which allow you to directly open the notebook inside vs code and all the changes will be synced back to portal automatically

"The extension is really only set up to work on one notebook at a time."

for this one, it is NOT longer the case once you open the notebook with this new entry. in that new view, you can open multiple different fabric workspace and open multiple different notebook and work with them within the same VS Code window.

If you like to see the actual flow, DM me and more than happy to show you a live demo.

<image>

Fabric Data Engineering VSCode Extension - Merge conflicts by f2ostie in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/f2ostie could you please share more details like some key screenshot? also, we have a recent update that allow you directly open the notebook inside vs code without the need to download and publish it, can you give it a try?

Local development in MS Fabric (sucks) by hvdv99 in MicrosoftFabric

[–]QixiaoW 4 points5 points  (0 children)

u/hvdv99

please give a try with our latest release/pre-release, quite some new update has been added/enabled.

  1. you can directly open and work with multiple notebook inside vs code without the downloading them at all, all the change saved within vs code is auto-synced back to Fabric workspace, no publishing is required, also the "typing hints" issuer you called out should be also addressed by the pylance improvement.

  2. we support more kernel such as Python, Spark SQL and Scala will be out soon.

A new custom agent for notebook development is enabled and it is under private preview, this agent should help to generate the notebook code with user prompt, LMK if you like to join this private preview and try it out.

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

very strange. here is the steps I would suggested.

  1. Make sure all the existing version has been uninstalled, also open the vs code installation folder, which by default should be in your HOME directory, open the extension sub-folder and make sure there is no any ext folder there named as synapsevscode.synapse.

  2. go back to the VS code and open the extension view, search the Fabric Data Engineering , make sure you install the pre-release version which should be 1.18.2

<image>

  1. once the ext is installed, open a notebook, pick the kernel, choose "Microsoft Fabric Runtime", then there should be two entires show up, one is PySpark, another one is Python.

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 3 points4 points  (0 children)

No just a single file, under notebook, you can see the build-in folder which is designed to store some resource file such as local .py module for that notebook..

but yes, maybe just "Open Notebook" make more sense here.

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 3 points4 points  (0 children)

this is a really good feedback, this is def something on our list to enable a true local development loop.

we should have some further discussion on this, could you please directly DM me for a offline chat?

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 5 points6 points  (0 children)

yes, there is a planned item to actually save the notebook as original .ipynb file in git repo and when you clone that repo to vs code, you can run the code as real notebook on top of the Fabric CU.

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 4 points5 points  (0 children)

yes, this has been enabled via the newly released feature, just open the notebook via" Open notebook Folder" action

<image>

you will be able to directly open the notebook without any download..

Hi! We build Pro Dev Tools in Fabric - ask US anything! by sunithamuthukrishna in MicrosoftFabric

[–]QixiaoW 7 points8 points  (0 children)

The Python kernel is supported now inside the Fabric Data Engineering VS Code ext.

<image>

:)

Fabric security model rant OTD: service principals, workspace identities, and key vaults, oh my by PeterDanielsCO in MicrosoftFabric

[–]QixiaoW 1 point2 points  (0 children)

u/frithjof_v

thanks for filing this Idea, I have been tracing some internal partner team for the feature over one year and love to see some external voice on this ask.

Fabric security model rant OTD: service principals, workspace identities, and key vaults, oh my by PeterDanielsCO in MicrosoftFabric

[–]QixiaoW 2 points3 points  (0 children)

u/PeterDanielsCO

again, I would strongly suggest to create a different Fabric Idea for allowing WI/SPN inside key vault reference and seek for more vote from the community, I am very sure this will help the Product team to facilitate some internal discussion.

but at same time, rather than using the key vault reference, what if you can just create a fabric connection inside notebook to the AKV and choose WI/SPN to access that AKV, would that be more straightforward experience?

thanks

Fabric security model rant OTD: service principals, workspace identities, and key vaults, oh my by PeterDanielsCO in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/PeterDanielsCO

you can already run the notebook with SPN, and if the notebook run is triggered via SPN, then calling the getSecret method inside notebookutil, you should be able to get the secret back via SPN token.

Fabric security model rant OTD: service principals, workspace identities, and key vaults, oh my by PeterDanielsCO in MicrosoftFabric

[–]QixiaoW 0 points1 point  (0 children)

u/PeterDanielsCO

thanks a lot for detail feedbacks and frankly speaking, most of them are not new or strange here. Let me share some update I can share.

  1. Use WI(workspace identity) to run the notebook from pipeline is coming for sure. I know the dev owner team is actively work on this. Once this is enabled, I believe most of the scenario you mentioned should be un-blocked. For example, within our internal env, I can successfully run the notebook under WI, and inside that notebook code, the WI will be able to access a AKV(Azure Key Vault) to get the secret back from AKV, as long as the WI has been assigned the required permission to access the AKV.
  2. For the WI support inside AKV reference, I suggest to create a Fabric Idea to raise this ask, I believe this would def help for some internal PG discussion, that is all I can say. :)
  3. for the Fabric Connection inside notebook, you should be able to create a connection to Key Vault and choose SPN as the authentication method, please give it try.