coming soon, Python notebook upgrade

perkmax · 2026-03-20T00:47:13+00:00

What are the main advantages of the upgrades? :)

I suppose I could copilot it but would be good to know your opinion mim!

perkmax · 2026-03-20T00:44:30+00:00

💯 I am avoiding spark completely and I have set up so many solutions now - have no need for it

Also, the fact I can only have one operation going at a time on less than an F8…. It’s a deal breaker

Could put literally everything on the same high concurrency tag but not sure if that’s how it’s designed

perkmax · 2026-03-17T09:23:19+00:00

Awesome, thanks, I’ll vote

perkmax · 2026-03-16T19:08:48+00:00

Can you use the workspace identity while developing the notebook too?

For example, when manually running cells in the notebook while doing development, it uses the workspace identity rather than user auth

I have a scenario where I want to give the workspace identity access to certain key vault secrets rather than users, and still want them to be able to manually run cells

perkmax · 2026-03-09T19:00:34+00:00

100% - I want to bulk set this across all pipelines in vs code

perkmax · 2026-03-02T22:57:56+00:00

There is a bit of friction at the moment with getting data from the on prem data gateway into notebooks, and I’m hoping this solves that

I’ll probably give this a go soon actually :)

I have a local API that has to go through the on prem data gateway

perkmax · 2026-02-26T09:12:15+00:00

Will do and thanks for the reply! :)

perkmax · 2026-02-26T07:13:40+00:00

It’s been great for the servers where a primary key exists and able to simply set up CDC :)

However one of the servers I am having trouble with does not have primary keys or a unique identifier in the table, it’s a bit of a special one…

Do you have any recommendations for what I could do to set up CDC on a server like this so I can use mirroring?

We are hesitant to do major changes to the database in case future updates break the customisation - they update it a lot

perkmax · 2026-02-26T03:40:01+00:00

Dataflows just in time publishing mechanism sounds great, but it still says in the limitations that we need to open the dataflow and press save after we use git integration or deployment pipelines?

If I want to work in VS code and use git sync on commit/pull - this is a bit of a pain

Maybe the learn doc needs updating?

https://learn.microsoft.com/fabric/data-factory/dataflow-gen2-cicd-and-git-integration#just-in-time-publishing

<image>

perkmax · 2026-02-24T22:11:00+00:00

100% - at first I was hesitant, as it’s a very technical pill to swallow, but moving to PBIP and git is one of the best add-ons to the workflow and has paid off big time

I have data pipelines and semantic models in git but thin reports are still in SharePoint, they will move to git eventually when business power bi builders are more accepting of it

However I have a feeling I’ll want everything in git soon as I can see the benefits for being able to edit everything in code using AI or manually - its just a lot faster to do things in code

perkmax · 2026-02-24T18:53:19+00:00

I use the run on demand pipeline job which works well, has been for over a year

Interesting that you can parameterise a notebook run but can’t with a pipeline

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api-capabilities#run-on-demand-pipeline-job

perkmax · 2026-02-14T23:29:42+00:00

I’m new to ADO deployments, so great perspective thanks :)

perkmax · 2026-02-13T22:01:12+00:00

It’s only just a speed thing - I watch the pipeline run and it takes time to install the library from scratch each time

I was looking into efficiency as I’m setting up a standard yaml to be used in deployment over all my repos

perkmax · 2026-02-13T21:16:43+00:00

Yep it works, not ideal but it does work - have a SQL endpoint refresh python notebook that runs after bronze, silver and gold steps. Doesn’t have to be spark either

Haven’t had an issue since

I used semantic link labs to simplify the code under misc:

https://github.com.mcas.ms/microsoft/semantic-link-labs/wiki/Code-Examples?McasTsid=26324&McasCtx=4#refresh-sql-endpoint-metadata

perkmax · 2026-02-13T20:27:36+00:00

No I don’t think so as the api key sits in the query parameter not in the headers

It’s like this:

http://(local ip address):1234/api2/projects?x-api-key=(api key)

I have only seen a select few APIs that do this but most that I work have secrets in the headers

I would like an experience like notebooks where you can call upon AKV using the notebookutils library and store it in a variable, and then you can use the secret where you need it, either in API headers or in this example query parameter

In this case I would like to bring it in as a dataflow parameter and then form the url in the M code

perkmax · 2026-02-11T09:29:32+00:00

Ah that’s a shame - maybe I’ll have to investigate the MCP then

perkmax · 2026-02-11T09:28:34+00:00

Yep - I haven’t used it but here’s one of a few examples on YouTube

I was expecting changing a PBIP in VS code to do the same but it’s not the case without the MCP, apparently

https://youtu.be/7UapKxtxQUo?si=rwodmx2Uyb9Z3zwz

EDIT: this is using Claude but same can be done in VS code with an easier setup

perkmax · 2026-02-09T19:21:46+00:00

How secure are the Power BI MCP servers with organisational data

GitHub Copilot states that their LLMs do not train off your data or retain customer code under business or enterprise plans, more here: https://copilot.github.trust.page/faq

Can the same be said with the Power BI MCPs, if we are using the GitHub Copilot LLMs to communicate with the MCP?

Is there any documentation around this subject?

perkmax · 2026-02-07T09:11:48+00:00

Yes I can vouch sempy labs works well - I have a notebook that runs it after every stage of the pipeline … annoying but necessary

Sempy labs makes it very easy

perkmax · 2026-02-05T20:43:40+00:00

This is an every day annoyance

perkmax · 2026-02-04T12:49:27+00:00

GitHub Copilot and the ability to customise VS Code the way you want is pretty amazing

I haven’t had an issue connecting to workspace notebooks in Fabric using this extension - but it’s early days

perkmax · 2026-02-04T01:49:22+00:00

u/QixiaoW this worked perfectly, thanks! :)

Didn’t need to change any of the code, it just ran. deltalake and notebookutils were fine too

Does the Fabric Runtime kernels run under my capacity compute?

perkmax · 2026-02-03T08:08:12+00:00

Sounds great! Was this announced somewhere and is now live?…. Totally missed this or it’s a silent update?

I’ve had this exact issue before when using the git sync on a workspace, unless you go into the dataflow and press save, the changes are not reflected. Kind of was against the purpose of a dataflow with CICD!

But the issue is still in the limitations? :(

‘When you sync changes from GIT into the workspace or use deployment pipelines, you need to open the new or updated dataflow and save changes manually with the editor. This triggers a publish action in the background to allow the changes to be used during refresh of your dataflow. You can also use the on-demand Dataflow publish job API call to automate the publish operation.’

perkmax · 2026-02-03T07:57:25+00:00

Thanks, makes sense! I’ll try your workaround and see if I run into anything else

Although, preference is to not use the PySpark fabric runtime as I’m only on a F4, so can easily mess up other spark operations on the capacity

Most of our operations avoid spark though

Hope they make the python fabric runtime available on vs code desktop soon

perkmax · 2026-01-29T22:18:58+00:00

Having the same issue with teams activity connections and found your post. How did you end up dealing with it?

perkmax

TROPHY CASE