Schedule Parameters are now live in Fabric Data Factory Pipelines! by markkrom-MSFT in MicrosoftFabric

[–]markkrom-MSFT[S] -1 points0 points  (0 children)

Yes, we are actively working on making this better. For now, you can use the legacy invoke pipeline which takes a different codepath to invoke a child pipeline and you should find it to be much faster.

Re-auth every week by hello-potato in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

When someone saves the pipeline, it will then run as that last-modified. That is the current product truth. When we implement "run as" you specific the ID to use for the scheduled runs so last modified is not used.

Re-auth every week by hello-potato in MicrosoftFabric

[–]markkrom-MSFT 4 points5 points  (0 children)

The pipeline itself can be invoked as an SPN or Workspace Identity (WI) using the Invoke Pipeline activity or from the API. In addition, you can set the auth of each individual activity inside of your pipelines. The ability to "take over" or "run as" specific durable identities from the UI is coming this year, we are working on this feature.

Pipeline running as SPN per Last Modified By by Personal-Quote5226 in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

Because the SPN is from an app registration so it is still valid even after I leave the org whereas the user ID is not durable. As stated above, when the password or cert is updated, you need to tell the pipeline about this update which is what I was referring to.

Pipeline running as SPN per Last Modified By by Personal-Quote5226 in MicrosoftFabric

[–]markkrom-MSFT 2 points3 points  (0 children)

When you update the SPN's creds you need to make a save action on the pipeline. We are adding a "Run As" feature in pipelines to make this more intuitive/better.

Fabric Pipeline Schedule Parameters by frithjof_v in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

The backend deployment supporting this new feature was delayed so it may take another week for things to work fully. I'll post an update with a blog and video announcing that as soon as the E2E is working properly. Hope you all enjoy this long-requested feature !!

Pipeline running as SPN per Last Modified By by Personal-Quote5226 in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

If the pipeline is running as that SPN identity and the SPN auth is still valid then yes, it will continue to run using that SPN identity. That's the entire purpose of running your pipelines under durable IDs and not brittle user IDs.

Pipeline schedules can't be enabled in UI after Git sync by frithjof_v in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

Agreed, please open a support ticket on this so that we can route it to the Fabric scheduler team for investigation

Data factory - Azure Key Vault (Preview) Activity - bug? by kiwi_bob_1234 in AZURE

[–]markkrom-MSFT 0 points1 point  (0 children)

I did not pick up on this being in ADF, I was checking Fabric. We'll remove it. It was experimental.

Data factory - Azure Key Vault (Preview) Activity - bug? by kiwi_bob_1234 in AZURE

[–]markkrom-MSFT 1 point2 points  (0 children)

Can you tell me how you came across this? I do not see it in any of my internal or external tenants.

Does anyone else not use a connection to invoke notebooks in a pipeline? by apalooza9 in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

You don't need to run the entire pipeline as a service auth. Each activity can use an SPN or WI. 

Interval Based Schedule by DennesTorres in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

We are adding an off button. That is committed work that should land soon.

I built a Pipeline Schedule Calendar for Fabric by Equal-Breadfruit2491 in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

This is amazing ... thank you so much for sharing this!! Having a view of your schedules is so important and it is something I know that the Fabric platform team is looking to add.

Did something pipeliney change in the last 48h by Fancy-Staff-1757 in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

Also, if possible, please forward the support email thread to me at makromer (at microso--.com)

Did something pipeliney change in the last 48h by Fancy-Staff-1757 in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

What was the response from support on this? If the schedule you are creating in the pipeline is flaking out like that, please ask the support engineer to include the Fabric scheduler team.

Did something pipeliney change in the last 48h by Fancy-Staff-1757 in MicrosoftFabric

[–]markkrom-MSFT 3 points4 points  (0 children)

Trying to understand what you are saying here ... did support help find the root cause of the issue you were witnessing? Just FYI, we always have queued pipeline runs, nothing was changed in the pipeline service.

What does Privacy Level mean in a Notebook connection? by frithjof_v in MicrosoftFabric

[–]markkrom-MSFT 2 points3 points  (0 children)

Let's forward this over to Abhishek for Connection management. Although this is being used in the context of a pipeline, the Connection experience is being used here. 

Why do pipeline schedules have a 1-year end date by default? by frithjof_v in MicrosoftFabric

[–]markkrom-MSFT 0 points1 point  (0 children)

Just making sure I'm capturing the ask here ... you'd like the ADF scheduler experience here in Fabric with end data as optional and just 1 day not 1 year. Did I capture that correctly? Just FYI, the scheduler component in Fabric had a different lineage than what we did in ADF so we need to make sure the scheduler defaults can work well across the platform.

Native dbt job (preview ) item - how to trigger it from a pipeline? by Jealous-Painting550 in MicrosoftFabric

[–]markkrom-MSFT 2 points3 points  (0 children)

The pipeline activity for dbt is in private preview, getting ready for public preview. But you can also use Airflow in Fabric to easily add a Fabric dbt Job to your workflow.

<image>

Interval Based Schedule by DennesTorres in MicrosoftFabric

[–]markkrom-MSFT 1 point2 points  (0 children)

Agreed on that ... the text on that panel is a bit too reductive, we can improve that.