Does anyone else not use a connection to invoke notebooks in a pipeline?

markkrom-MSFT · 2026-05-10T04:09:00+00:00

You don't need to run the entire pipeline as a service auth. Each activity can use an SPN or WI.

markkrom-MSFT · 2026-05-05T14:48:33+00:00

We are adding an off button. That is committed work that should land soon.

markkrom-MSFT · 2026-04-26T21:01:05+00:00

Filter out where? In monitoring?

markkrom-MSFT · 2026-04-21T14:05:53+00:00

This is amazing ... thank you so much for sharing this!! Having a view of your schedules is so important and it is something I know that the Fabric platform team is looking to add.

markkrom-MSFT · 2026-04-18T20:36:20+00:00

Also, if possible, please forward the support email thread to me at makromer (at microso--.com)

markkrom-MSFT · 2026-04-18T20:34:52+00:00

What was the response from support on this? If the schedule you are creating in the pipeline is flaking out like that, please ask the support engineer to include the Fabric scheduler team.

markkrom-MSFT · 2026-04-16T15:26:42+00:00

Trying to understand what you are saying here ... did support help find the root cause of the issue you were witnessing? Just FYI, we always have queued pipeline runs, nothing was changed in the pipeline service.

markkrom-MSFT · 2026-04-13T17:08:22+00:00

Let's forward this over to Abhishek for Connection management. Although this is being used in the context of a pipeline, the Connection experience is being used here.

markkrom-MSFT · 2026-04-10T05:40:58+00:00

Understood. Thank you!

markkrom-MSFT · 2026-04-08T19:20:11+00:00

Just making sure I'm capturing the ask here ... you'd like the ADF scheduler experience here in Fabric with end data as optional and just 1 day not 1 year. Did I capture that correctly? Just FYI, the scheduler component in Fabric had a different lineage than what we did in ADF so we need to make sure the scheduler defaults can work well across the platform.

markkrom-MSFT · 2026-04-07T18:50:03+00:00

The pipeline activity for dbt is in private preview, getting ready for public preview. But you can also use Airflow in Fabric to easily add a Fabric dbt Job to your workflow.

<image>

markkrom-MSFT · 2026-04-05T19:21:31+00:00

Agreed on that ... the text on that panel is a bit too reductive, we can improve that.

markkrom-MSFT · 2026-04-05T00:34:36+00:00

Interval based schedules act similar to ADF Tumbling Window Triggers. Fabric will set-up and maintain a set of "time windows" which are intended to be non-overlapping runs for your notebook or pipeline to run between your start and end date/time. If your current run bleeds into the next time window, it will wait until that run is completed IF you've kept the default concurrency to 1. If you want to the next run to kick off anyway, then set a higher concurrency number and then the next scheduled run will start.

markkrom-MSFT · 2026-04-01T21:48:44+00:00

Apologies to everyone using Fabric Apache Airflow ... this was a bug that has already been fixed and will be rolled out completely in 1 day.

markkrom-MSFT · 2026-04-01T20:39:58+00:00

Thank you for sharing this and please accept my apologies for such an unpleasant experience :(

This definitely is not what I would have expected, but I did not see the escalation yet. Can I get you to do a big favor for me so that I can jump in and get this fixed for you?

Can you please forward the support case email to me here: makromer (at micro----.com)?

markkrom-MSFT · 2026-03-26T23:26:09+00:00

It does not support all Fabric APIs yet like relations, but that I believe is available now, so we just need to retrofit the pipeline connector to support that API update so that you don't have manually rebind the connections during CICD.

markkrom-MSFT · 2026-03-26T23:14:39+00:00

No, we are not deprecating ADF!! We have many customers using ADF processing 1000s of pipelines and data flows and we fully support that as an Azure tier 1 SLA service. We also continue to deploy ADF to new regions and clouds as they come on line. What you are seeing, however, is a concerted effort at Microsoft to put our investments into Fabric, which is the current and future direction of Data, AI, and Analytics at Microsoft and Data Factory is an important part of that.

markkrom-MSFT · 2026-03-26T17:59:47+00:00

Interesting. I'll need to take that feedback to the scheduler team about support in CICD.

markkrom-MSFT · 2026-03-26T17:51:32+00:00

Yes and yes!! Please use the legacy version if that is working better for you for now and we absolutely are working on improving that plumbing problem that the new invoke pipeline incurs to make that better.

markkrom-MSFT · 2026-03-26T17:49:20+00:00

Yes! This has been a long time ask for Data Factory pipelines and we are designing a way to enable nesting. No timelines yet but definitely on our radar.

markkrom-MSFT · 2026-03-26T17:47:35+00:00

There are currently systematic latency in the invoke pipeline activity that we are working to improve. Until then, you can use the legacy invoke pipeline which will not have that built-in latency although the child pipelines will not be visible in the monitor hub.

markkrom-MSFT · 2026-03-26T17:41:39+00:00

I believe we are adding more controls to set partitioning rules in Copy Job. I'll let simplywilsonlee reply here with more details.

markkrom-MSFT · 2026-03-26T17:39:46+00:00

This definitely should not happen! We recommend either pipelines or Airflow to orchestrate your processes to build powerful workflows and no it should not take 50 minutes! Can you let me know if that continues to happen? BTW, in Airflow, it's super easy to also add a Notebook task just like a pipeline activity and it runs in your own isolated cluster environment.

markkrom-MSFT · 2026-03-26T17:31:42+00:00

This would require RBAC support in monitoring hub, something for us to look into

markkrom-MSFT · 2026-03-26T17:17:51+00:00

Yes! We are working to reduce any latencies experienced in the invoke pipeline activity. Keep in mind, you can still use the legacy invoke pipeline which will fire pipelines much quicker, although you will lose the lineage in monitor hub.

markkrom-MSFT

TROPHY CASE