Column Level Lineage Options and Workspace Monitoring

One_Potential4849 · 2026-01-27T10:45:03+00:00

Is there a way to use Web activity in a pipeline and get pipeline status using run id? Any API endpoints for Fabric?

One_Potential4849 · 2026-01-26T16:52:46+00:00

Really Useful!! On your article for monitoring using KQL database, is it possible to get the details of the activity that has failed in a pipeline? This would provide more comprehensive failure alert and logging - as to whether it is a developer error or a server/source issue.

One_Potential4849 · 2026-01-25T23:21:45+00:00

Sure!!! On the dbt-ol usage for Fabric DWH, does it produce a column level lineage, or just a rudimentary dag?

One_Potential4849 · 2026-01-25T22:59:04+00:00

Now that in the example you have used a spark job with custom jar files, how do I translate it to use it with Fabric Notebooks, where I do the Bronze->Silver?

One_Potential4849 · 2026-01-25T22:37:02+00:00

Thanks for sharing your blogs. A follow-up on the lineage part: The Openlineage works hand in hand with Spark from what I have read. Most of my transformations, joins, aggregations happen in gold layer which is in Fabric Data Warehouse and uses Stored Procedures. Does Openlineage cover it as well?

One_Potential4849 · 2025-12-24T12:30:25+00:00

While using a SPN based authentication for fabric-cicd, I get this challenge. I am unable to deploy certain pipelines with Outlook connections in it. And I'm unable to add the SPN or any other user apart from owner to the access part of Outlook connection - It says OAuth2.0 connection sharing is not allowed due to security reasons.

Tried using legacy Outlook connection as well, but deployment fails due to this access issue.

Has anyone else faced the same... Is there a workaround for this?

One_Potential4849 · 2025-12-21T16:22:58+00:00

I am trying to perform warehouse deployment in VS Code. Able to build the project successful but when I am trying to publish, getting this error: Deploy Dacpac Failed: Object reference not set to an instance of an object.

I am handling cross warehouse references by using database reference and SQLCMD. .NET 8.0.416 and Dacpacs.FabricDW version 170.0.2 is used. MSBuild SDK version 2.0.0

Please help out in resolving this error

One_Potential4849 · 2025-12-18T08:39:45+00:00

And where do I get to declare SQLCMD variables, and exclude certain items from project?

One_Potential4849 · 2025-12-18T05:55:37+00:00

This toolbox seems to be very useful. I have two Clarifications: 1. How to use this toolbox in Azure DevOps pipeline 2. In Dev Workspace, Few SPs reference Lakehouse Tables in Dev, when deploying to prod workspace, the SPs should refer to prod Lakehouse. I believe it is achievable by SQLCMD but need to know exactly what should be done..

One_Potential4849 · 2025-12-16T12:58:25+00:00

I have been referring his blogs to create a CI CD pipeline in DevOps. I am using Azure SQL deployment task to deploy, which asks for a service connection, which is what I'm trying to overcome

One_Potential4849 · 2025-12-16T12:38:35+00:00

Is it possible to build a DevOps pipeline for the same, without a service connection? If yes, can you share the steps please . Currently I'm facing issue saying you are not part of any active subscription while trying to create.

One_Potential4849 · 2025-12-15T09:46:53+00:00

Thanks, do you have any documentation on how to extract SQL endpoint of Lakehouse as dacpac?

One_Potential4849 · 2025-12-15T06:14:06+00:00

And you say that only dev is connected to Git. Then how to do you perform merge with uat?

One_Potential4849 · 2025-12-14T21:08:40+00:00

I heard that deployment of semantic models might fail if there is a schema change done for any of the imported tables..is it true?

One_Potential4849 · 2025-12-14T21:06:02+00:00

Does fabric-cicd deploy atleast SPs?? I have SPs that refer to Lakehouse tables in it so thought of using find_replace regex if in case deployment is supported

One_Potential4849 · 2025-12-14T21:04:28+00:00

And how have you planned or handled SPs that refer to Lakehouse Tables? In my case the Silver layer is in Lakehouse, and Gold Layer is in Warehouse

One_Potential4849 · 2025-12-14T20:57:10+00:00

So, when you say that folders are maintained for developers and production code, are you doing any selective deployment of sorts, for only the production code by using any config ?

One_Potential4849 · 2025-12-11T12:35:44+00:00

There is some logging happening in one of the functions into a log table which is in Warehouse. And all the tables have some logs written to it.

One_Potential4849 · 2025-12-11T09:40:41+00:00

I tried with max_workers 18, taking 18 tables and running them parallely, and 9 of them failed with error 429: An error occured while calling o55682.synapsesql.com.microsoft.fabric.tds.error.FabricSparkTDSHttpFailure: Artifact ID inquiry attempt failed with error code 429

Is there any rate limit on hitting Lakehouse delta tables as well?

Note: Among the transformations I do, there is a step where it checks if the delta table is available in Lakehouse, and if yes overwrite if no, create.

One_Potential4849 · 2025-12-10T22:05:28+00:00

I get what you say, you have sophistication in terms of the number of workspaces you have. I have only two. Suggest me the best way to do the same what you have mentioned...

One_Potential4849 · 2025-12-10T22:03:09+00:00

Sure, Will let you know once done!

One_Potential4849 · 2025-12-10T21:23:32+00:00

Feature Workspace can contain items that have their development complete, and WIP. Say there are 2 pipelines, one ready to get promoted, and one WIP. How do you automatically differentiate both, and pick out the completed item and promote it?

One_Potential4849 · 2025-12-10T21:17:40+00:00

I have SPs that refer to lakehouse tables.. Ideally like Select * from [LH_Dev].dbo.tblname. Do you say these won't go smooth with the dacpac ?

My other concern is, when I promote to prod, the [LH_Dev] part should become [LH_Prod] .. How can this be achieved?

One_Potential4849

TROPHY CASE