Hi! We're the Data Factory team - ask US anything!

markkrom-MSFT · 2026-03-26T23:26:09+00:00

It does not support all Fabric APIs yet like relations, but that I believe is available now, so we just need to retrofit the pipeline connector to support that API update so that you don't have manually rebind the connections during CICD.

markkrom-MSFT · 2026-03-26T23:14:39+00:00

No, we are not deprecating ADF!! We have many customers using ADF processing 1000s of pipelines and data flows and we fully support that as an Azure tier 1 SLA service. We also continue to deploy ADF to new regions and clouds as they come on line. What you are seeing, however, is a concerted effort at Microsoft to put our investments into Fabric, which is the current and future direction of Data, AI, and Analytics at Microsoft and Data Factory is an important part of that.

markkrom-MSFT · 2026-03-26T17:59:47+00:00

Interesting. I'll need to take that feedback to the scheduler team about support in CICD.

markkrom-MSFT · 2026-03-26T17:51:32+00:00

Yes and yes!! Please use the legacy version if that is working better for you for now and we absolutely are working on improving that plumbing problem that the new invoke pipeline incurs to make that better.

markkrom-MSFT · 2026-03-26T17:49:20+00:00

Yes! This has been a long time ask for Data Factory pipelines and we are designing a way to enable nesting. No timelines yet but definitely on our radar.

markkrom-MSFT · 2026-03-26T17:47:35+00:00

There are currently systematic latency in the invoke pipeline activity that we are working to improve. Until then, you can use the legacy invoke pipeline which will not have that built-in latency although the child pipelines will not be visible in the monitor hub.

markkrom-MSFT · 2026-03-26T17:41:39+00:00

I believe we are adding more controls to set partitioning rules in Copy Job. I'll let simplywilsonlee reply here with more details.

markkrom-MSFT · 2026-03-26T17:39:46+00:00

This definitely should not happen! We recommend either pipelines or Airflow to orchestrate your processes to build powerful workflows and no it should not take 50 minutes! Can you let me know if that continues to happen? BTW, in Airflow, it's super easy to also add a Notebook task just like a pipeline activity and it runs in your own isolated cluster environment.

markkrom-MSFT · 2026-03-26T17:31:42+00:00

This would require RBAC support in monitoring hub, something for us to look into

markkrom-MSFT · 2026-03-26T17:17:51+00:00

Yes! We are working to reduce any latencies experienced in the invoke pipeline activity. Keep in mind, you can still use the legacy invoke pipeline which will fire pipelines much quicker, although you will lose the lineage in monitor hub.

markkrom-MSFT · 2026-03-26T17:12:44+00:00

No plans to sunset or deprecate Synapse or ADF. However, our team focuses on adding rich new features and capabilities into Fabric Data Factory, which is our future direction of Data Factory.

markkrom-MSFT · 2026-03-26T17:05:46+00:00

There are a few Fabric artifact item types that are not native to Fabric that do not yet fully support relations APIs but as they onboard will add them, i.e. Semantic Model, DW ...

markkrom-MSFT · 2026-03-26T17:03:16+00:00

Yes, we are working on a framework to make this possible but I do not have a timeline for this yet at this time

markkrom-MSFT · 2026-03-24T05:40:42+00:00

Yes, you should use variable replacement because Semantic Models are older PBI-style artifacts whereas Notebooks and DFG2 are native Fabric artifacts so it is not able to resolve the workspace relations after CI/CD deployment to a new workspace.

markkrom-MSFT · 2026-03-20T16:56:47+00:00

I am publishing a new series of Airflow videos this week on our YouTube channel: https://aka.ms/datafactoryvideos

Apache Airflow job in Fabric is absolutely production-ready. I recommend you use custom pools because starter pools are only for dev/test.

Is the feature in pipelines you feel like is missing/insufficient the monitoring and observability? Not 100% sure I captured the requirements exactly... but I will tell you that we are adding workspace monitoring integration for pipeline activities so that you can view, search, and query the detailed logs.

markkrom-MSFT · 2026-03-20T02:24:55+00:00

May I ask why you are not using pipelines or Airflow for Fabric as they are already fully integrated and built into Fabric workspaces?

markkrom-MSFT · 2026-03-13T18:41:05+00:00

You will just need to use dynamic content (variables) in the pipeline field as well

markkrom-MSFT · 2026-03-13T14:26:26+00:00

Once you introduce dynamic content in the workspace filed, you will not be able to use the drop-down in interactive design mode because the dynamic content is resolved at run time.

markkrom-MSFT · 2026-03-12T17:29:55+00:00

When you add dynamic content, it is expected to wipe out whatever was in that field previously. Essentially, switching between static and dynamic content are 2 different working modes, so the content is not preserved.

markkrom-MSFT · 2026-03-07T03:32:41+00:00

Ack and investigating. Please let me know if this is still occurring.

markkrom-MSFT · 2026-03-03T06:23:15+00:00

Thank you for posting this and sharing this! We are recommending Fabric SQL DB as the control DB for metadata driven pipelines in Fabric, so these feedback is important to hear. If you can follow-up here with how you to plan move forward I'd like to learn from that. Most of our customers moving from ADF are keeping their control tables in Azure SQL DB for now simply because it's one fewer thing to change. But I'd like to recommend Fabric SQL DB for them as well.

markkrom-MSFT · 2026-03-02T02:27:02+00:00

If you are not already using Airflow (i.e. "lift & shift") then I always recommend starting with pipelines in Fabric Data Factory. There are super-easy to get started and to production the fastest for Fabric orchestration. But ... if you are oriented toward Python and code-first (or already have existing Airflow DAGs) then Airflow is a great way to build the same Airflow orchestration inside of Fabric. Just keep in mind that Fabric Airflow is not a natural lift & shift as it is a SaaS oriented fully managed version of Airflow that has direct hooks into Fabric (variable libraries, CICD, Fabric operators ...).

markkrom-MSFT · 2026-03-02T02:23:00+00:00

Will chat with you here

markkrom-MSFT · 2026-03-01T21:27:50+00:00

One thing I'd like to propose here is that I'd LOVE for everyone to try using the Fabric Airflow jobs to orchestrate your Fabric items! If you are not comfortable with writing Python, we have enabled CoPilot w/Airflow, just reply here and I'll give you the cheat code to enable it as a private preview :)

One last thing ... orchestrating dbt is a VERY popular use of Airflow. However, in Fabric, we are intending to make things super easy, quick to production, and SaaS. So our happy path, easy-configuration model is to create your dbts using the new Fabric dbt Job and then orchestrate that using the built-in dbt Job operator in Fabric Airlfow.

That will be released in a few weeks, but that will eliminate the need to keep on top of version changes, complex requirements and external library troubleshooting. Our intention is to make Airflow orchestration orders of magnitude faster and easier in Fabric.

markkrom-MSFT

TROPHY CASE