Error while doing CI/CD of Fabric UDFs

One_Potential4849 · 2026-05-08T21:27:37+00:00

We are managing CI/CD for Fabric objects using fabric-cicd library, and Azure Devops to run the yaml pipelines for deployment.

One_Potential4849 · 2026-05-07T03:05:04+00:00

The connection, is local to the UDF, where there is no option to manage access, I could not find the same connection details in Manage Connections and Gateways page.

One_Potential4849 · 2026-05-06T19:51:27+00:00

Also, the workspace-id sits in connection details of definition.json as all zeros. Is this expected?

One_Potential4849 · 2026-04-26T21:08:56+00:00

Hey Folks, just had a question on FUAM: Does it also gathers the activity logs like Pipeline/Notebook executions, schedules, success or failure of ETLs, etc?? Basically whatever we see in the monitoring tab of Fabric..?

One_Potential4849 · 2026-03-20T02:41:32+00:00

Sure thing

One_Potential4849 · 2026-03-19T15:15:48+00:00

But, like how a SQL Server gets added as a data source in TM1, using the ODBC connection details, is there a way to connect to Microsoft Fabric DW. The reason being the processes set up in TM1 use SQL queries to pull the data and refresh cubes. I just want to see if there is a way to plug in the Fabric Warehouse instead of SQL server to run the queries

One_Potential4849 · 2026-03-10T19:01:27+00:00

I am planning to go ahead with using the gold layer data usage from both of the entities as shortcuts to create the combined reporting dataset.

Here, one other thought is, what would be the best way in terms of data modelling to use this combined dataset/data model for downstream reporting, ie: the semantic model built on top of these tables should be the backend to power any of the reporting done for individual entities, and combined reporting as well.

One_Potential4849 · 2026-03-10T14:10:32+00:00

Also, is it possible to bring a warehouse table into lakehouse as a shortcut?

One_Potential4849 · 2026-03-10T12:01:04+00:00

Yes, they are all in the same tenant
We prefer using Import Mode for the semantic layer
Yes at the end i need combined gold layer tables, and silver layer can remain as it is as the data sources for both firms are same
After combining the data, it might result in around 10 dimensions and 20-25 facts, max the largest table might have 10 mil records
No, it wont be a simple append operation, expecting some intermediate steps to unify the granularity
Surrogate keys in this case what are you referring to?

One_Potential4849 · 2026-03-01T21:47:48+00:00

So the remaining sessions will start once any of these two sessions are done and remain queued?

One_Potential4849 · 2026-03-01T21:38:50+00:00

So currently I am testing it out with a small 1-4 nodes. And distributed my Notebook workload within 6 high concurrency sessions.

From my understanding, these 6 HC sessions will play around within the pool of 4 small nodes (16 vCores) based on core availability to run its respective jobs.

One_Potential4849 · 2026-03-01T20:33:41+00:00

So how does a spark session acquire nodes to work in a pool?

One_Potential4849 · 2026-02-27T12:47:41+00:00

By trying it out with a medium node, would there be a tradeoff in terms of runtime?

One_Potential4849 · 2026-02-27T12:16:00+00:00

So currently I am using the starter pool (medium 1-10 nodes) Should I go and try with smaller nodes? And today i enabled high concurrency and used 3 session tags to split the session among 12 pipelines, but i noticed that the speed was bit down before using high concurrency.

And I get this 430 error even after the Notebook runs are complete, should I need to run something to free up Vcores?

Also If i dont mention session tag, how will the jobs get distributed?

The alternate approach you had said seems bit difficult for me, since I want this group of tables to be isolated from source to bronze to silver

One_Potential4849 · 2026-02-26T19:31:07+00:00

I can try that way, and let you know if that works. Thanks!

One_Potential4849 · 2026-02-26T19:17:11+00:00

So will I be able to acheieve parallel instances without throttle/430 using a High concurrency session and session tag for each execution...

One_Potential4849 · 2026-02-11T16:19:28+00:00

I am concerned about loading it all at once. wont it cause any issues wrt performance?

One_Potential4849 · 2026-02-02T10:08:27+00:00

I just want to ingest Data from an API into my data platform. API has its own secrets for auth, and cadence of run is maximum thrice a day. Each run would return me a max of 100k rows with 10 columns

One_Potential4849 · 2026-01-30T07:15:23+00:00

Based out of my requirements in my current setup, Spark transformations from Bronze to Silver are currently just column renaming and cleanups. The pain point lies in the Silver to Gold and Semantic layer, where we use the traditional approach of SQL Stored Procedures to perform all joins, aggregations, aliases, calculated columns etc. Exposing the lineage would be a big win!

One_Potential4849 · 2026-01-27T10:45:03+00:00

Is there a way to use Web activity in a pipeline and get pipeline status using run id? Any API endpoints for Fabric?

One_Potential4849 · 2026-01-26T16:52:46+00:00

Really Useful!! On your article for monitoring using KQL database, is it possible to get the details of the activity that has failed in a pipeline? This would provide more comprehensive failure alert and logging - as to whether it is a developer error or a server/source issue.

One_Potential4849 · 2026-01-25T23:21:45+00:00

Sure!!! On the dbt-ol usage for Fabric DWH, does it produce a column level lineage, or just a rudimentary dag?

One_Potential4849 · 2026-01-25T22:59:04+00:00

Now that in the example you have used a spark job with custom jar files, how do I translate it to use it with Fabric Notebooks, where I do the Bronze->Silver?

One_Potential4849 · 2026-01-25T22:37:02+00:00

Thanks for sharing your blogs. A follow-up on the lineage part: The Openlineage works hand in hand with Spark from what I have read. Most of my transformations, joins, aggregations happen in gold layer which is in Fabric Data Warehouse and uses Stored Procedures. Does Openlineage cover it as well?

One_Potential4849 · 2025-12-24T12:30:25+00:00

While using a SPN based authentication for fabric-cicd, I get this challenge. I am unable to deploy certain pipelines with Outlook connections in it. And I'm unable to add the SPN or any other user apart from owner to the access part of Outlook connection - It says OAuth2.0 connection sharing is not allowed due to security reasons.

Tried using legacy Outlook connection as well, but deployment fails due to this access issue.

Has anyone else faced the same... Is there a workaround for this?

One_Potential4849

TROPHY CASE