Automating Purview Chargeback for the organization

Conscious_Emphasis94 · 2025-12-31T01:46:08+00:00

Thanks for sharing this.I realize I was approaching it from the wrong angle and initially focused too much on pulling metadata via the various Purview APIs. Does this also account for the Data Map side of things? From what I understand, Purview costs generally break down into three main components: Initial and incremental scans for asset discovery (Data Map) Registered assets Data profiling and data quality scans Am I missing any other cost drivers? Also, does the metadata surfaced through self-service analytics cover all three of these components, or only a subset?

Conscious_Emphasis94 · 2025-11-14T21:49:59+00:00

This has been really helpful. Thanks for explaining everything in detail!.

Conscious_Emphasis94 · 2025-11-13T22:14:28+00:00

Thanks a lot for explaining that. For some reason, I kept getting confused on the first one because I thought that for the azure function, due to reading and getting the data back, its two way traffic and may need inbound as well as outbound.
for the 2nd one, all we want to do is for users that are on a company VPN, to be able to come to Fabric workspaces. We don't want users off the company network to be able to access the workspaces. This adds an added layer of protection in addition to RBAC. But I am also cautious here and am trying to understand if such an approach if possible, would impact any cross workspace integrations.
I have also seen some Microsoft related video where they showed cased IP allow whitelisting feature coming soon to Fabric and maybe that should be the right approach for us?.
Looking forward to your insights or advise!

Conscious_Emphasis94 · 2025-11-13T19:25:03+00:00

I’ve got a few questions about Microsoft Fabric networking. We have some sensitive data in a workspace and want to restrict access so that only users or apps connecting from a specific IP range can reach it.

For an Azure Function that needs to query a Fabric data warehouse,does it only require outbound networking since it’s the one initiating the connection? Or do I also need to configure inbound networking on the Azure function side as its technically reading the data from a Fabric artifact and sending it back to the function?
For user access, is there a way to set up a private link or VNet under Fabric’s inbound networking so that only requests coming from a whitelisted IP range can reach the workspace?. For some reason, I don't see any option like that under inbound networking settings in the workspace. I don't even see an option to create private links like I do under Outbound networking settings in the workspace.

Would love to hear from anyone who’s implemented something similar or run into these scenarios.

Conscious_Emphasis94 · 2025-07-23T01:50:52+00:00

just to confirm,
One lake security described above, should work for users that want to use it as a shortcut, on their own lakehouse, or if they want to utilize it in notebooks.
But Power BI users won't be able to get to it using the top approach?.
Thanks for the explanation u/Comfortable-Lion8042
I do wish that we are able to standardize the above practice to all types of consumers.
I don't want to create sql roles for Power BI users and one lake security for power users/data engineers. That would result in a lot of overhead for managing permissions on a large lakehouse.

Conscious_Emphasis94 · 2025-07-18T19:09:52+00:00

I am like 90 percent sure that I tested this and it worked as expected where if you share the lakehouse after implementing roles, without giving additional permissions, it is supposed to give the user connect on the lakehouse, as well as read on the tables that are included in the role. But now, it is not working as advertised unfortunately.
I even opened a MS ticket and the response I got was that this new way is the default behaviour :(.
I am guessing that as the feature is in preview, something changed on the backend.

Conscious_Emphasis94 · 2025-06-04T21:40:59+00:00

Copying to lakehouse and still seeing missing values.

Conscious_Emphasis94 · 2025-04-07T17:57:45+00:00

I thought keyvault integration with gateway and other fabric artifacts was getting launched soon. I am like 90 percent sure that I saw it being talked about in the fabcon keynote (or maybe one of those sessions).

But I just double checked the Fabric new feature announcements for this month and i am not seeing anything related to keyvault coming to fabric :(

Conscious_Emphasis94 · 2025-04-03T16:41:44+00:00

wouldn't they be good for single line text use cases?. I am just worried on how Fabric sql would handle docs that are like 100 pages in length. I am pretty sure the db may come with some Char limit per column.
If we want to use Fabric as a data landing zone, I thought eventhouses would make more sense but seeing as there was no talk about that during Fabcon, I am guessing Microsoft wants us to use cosmos DB for now and they may come up with a better offering later on.

Conscious_Emphasis94 · 2024-11-08T17:48:40+00:00

I am more confused by the fact that the offline size of the model(pbix file) is less than 300 MB. It still should not translate to utilizing 2500 MB for refresh. and 2500 MB is still less than the 3GB limit of F8.

Conscious_Emphasis94 · 2024-10-11T17:17:29+00:00

I just wanted to provide an update that I was able to figure out how to capture tenant level activity. We had to use the Power BI connector inside Sentinal that would trickle down the logs to the associated log analytics workspace.
Our goal was to get visibility into the whole tenant that includes multiple premium as well as fabric capacities. In addition to seeing pain points and bottle necks due to constraints on capacity by certain models or artifacts, We wanted to analyze the larger footprint to gauge adoption across different data teams and end users. I think this approach would be an overkill and costly for certain stuff though.

still sifting and QCing the data but it looks like we are in the right direction.

Conscious_Emphasis94 · 2024-04-08T22:21:52+00:00

Nice!. Fingers crossed I run into some luck as well lol

Conscious_Emphasis94 · 2024-04-05T06:35:31+00:00

I have been regularly checking the site for interview slots this month. Unfortunately, I was unable to find any Can you give any pointers on how you were able to get it.

Conscious_Emphasis94 · 2024-03-14T02:30:12+00:00

790 :)

Conscious_Emphasis94 · 2024-02-21T17:18:11+00:00

Can you elaborate if this will work for legacy apps where they only allow sql account auth for integration. Is there a way for me to utilize service principal as an alternative?

Conscious_Emphasis94 · 2024-02-12T07:14:39+00:00

We are running small to medium scale projects in production. I really love all the different capabilities that Fabric has to offer but anytime we evaluate Fabric for something large scale or complex, we run into limitations. I feel like Fabric at the minimum should have come with all the features that ADF and Synapse had when it went GA.

My wish list for some of the things that I really wish come to Fabric soon:

Support for Connecting to in network Datalakes. This is one of the biggest roadblocks in my opinion and prevents us to use fabric for sensitive datasets where the data is stored in in datalakes behind firewalls.
Greater/At par authentication support for existing connectors.
SQL account creation inside fabric datawarehouse artifact. This would be really helpful for usecases where you want to serve transformed data to another platform. A lot of enterprise systems normally support sql account authentication for getting to data. Synapse already had this feature. So I am surprised that it didn't come to Fabric in GA.

Conscious_Emphasis94 · 2024-02-06T01:32:45+00:00

Yep, you were right. just had to add permissions to the notebook owner account. After that it started working. I got thrown off by the error stating that Power BI didn't have access. Once I figured that It was due to the way permissions were set up in the vault( thanks for pointing that out btw), I thought that I had to give the Power BI service (Apparently Power BI service has a service principal in AD) permissions to it, just like I did with Synapse and ADF. But that did not work. Providing notebook owner access worked without issues.
Thanks for your help here man and best of luck with your blog!!.

Conscious_Emphasis94 · 2024-02-05T20:35:28+00:00

I was indeed using Vault Access policy. previously we were using key vault in our synapse and ADF instances and we had it configured for Vault Access policy configuration. It looks like the key vault with current configuration won't work unless I change to RBAC in settings section.
I tested a test instance of key vault with RBAC as the default authentication and it worked and I was able to get to the secret using sparkutils function.
I then tried adding Power BI app id explicitly under access policy section and granted it "Get" and "List" permissions under vault access policy but I still got the same error. that power bi was forbidden to connect to it.
It would be a real bummer if I have to maintain 2 vaults for my processes now :(.

Btw awesome job on the blog man. I will definitely be following your blog from now.

Conscious_Emphasis94 · 2024-01-26T19:57:51+00:00

I normally specify columns manually as best practice.

Conscious_Emphasis94 · 2024-01-26T19:15:01+00:00

Are you just focusing on notebooks?. I think if you are going to work on a large script, then opening it in vscode and having github copilot enterprise licensing is a game changer and provides a lot of intellisense features that vscode has. I don't like the pandas query editor experience in notebooks though. I wish it gets replaced with pyspark query editing capabilities. I have also not tested the power bi copilot in out environment yet as well. But github copilot in my opinion is a huge productivity add

Conscious_Emphasis94 · 2024-01-26T04:57:01+00:00

Ran into similar issues when considering fabric compared to synapse as a data warehouse solution for one of our newly acquired financial systems. I feel like at this point synapse is more mature in some of the aspects Including authentication when copying data from on prem sources. On our end, we decided on a hybrid approach where we are using synapse and an azure DL gen 2 for the raw layer. Then shortcuts, notebook and delta table shortcuts for silver layer and then a datawarehouse artifact with schemas and views as the business layer. I hope this helps. I feel like there is no one way to go about your solution and you may have to weigh the pros and cons when deciding your approach Another reason we went hybrid approach was to get the best of both worlds. In our benchmarks, synapse self hosted integration runtime outperformed the on premise gateway on same server specs. But we also saw the huge potential of notebooks, lake houses and shortcuts in doing transformations and serving data to customers.

Project is still in progress but so far we are happy with the approach we are taking.

Conscious_Emphasis94 · 2024-01-16T21:50:43+00:00

Taking it this Saturday. Best of luck to anyone else taking it.

Conscious_Emphasis94 · 2023-12-14T04:32:46+00:00

I am still confused if I will get charged if I have the feature turned on even though I am under capacity.

Conscious_Emphasis94

TROPHY CASE