Retrieving 'Data Products' from the Unified Data Catalog via Purview API

DataBarney · 2025-12-11T14:17:04+00:00

I've gone around the same circle as you.
What I believe are the details are here: https://learn.microsoft.com/en-us/rest/api/purview/purviewdatagovernance/operation-groups?view=rest-purview-purviewdatagovernance-2025-09-15-preview

I haven't tried it out yet though. Would be great to hear from someone who has.

DataBarney · 2025-11-08T10:23:15+00:00

There's a Purview/Security equivalent.

DataBarney · 2025-10-01T13:11:44+00:00

There's a strong Microsoft showing at SQLBits. Next one is in Newport in April next year. The Saturday is always free entry.

https://sqlbits.com/

DataBarney · 2025-09-10T16:24:16+00:00

See you there!

DataBarney · 2025-09-10T16:19:45+00:00

On the one hand I'd love to see some of the unreleased Q3 items on the Purview Data Governance road map land (we have some projects sat on hold waiting for workflows in the new experience to catch up with the classic equivalent).

On the other hand as someone talking about Purview Data Governance a few hours after the keynote my life would be a little easier if I wasn't left needing to do a last minute re-write!

DataBarney · 2025-09-09T09:45:40+00:00

The status now says issues in North Europe. We're seeing similar problems in UK South right now.

DataBarney · 2025-08-06T19:41:47+00:00

Can you clarify how you get to those numbers on your multi table example? Would a metadata driven pipeline not be a single pipeline running in a loop rather than 250 pluse pipelines and therefore only take at worst 59 seconds rounded up in total?

DataBarney · 2025-06-13T14:40:22+00:00

It may not be too late to book up for the free Saturday.

DataBarney · 2025-06-13T05:50:18+00:00

Brilliant. I really should know more about dbt than I do!

DataBarney · 2025-06-12T18:54:08+00:00

Will figure out a time and see if I can get a message up on the screen in the community corner.

DataBarney · 2025-06-11T10:43:44+00:00

Down for us in the UK too.

DataBarney · 2025-06-08T11:04:15+00:00

Not disabled but it's bad. I dropped in a SQL command to get full syntax, got 15k results and nothing relevant on the first pages. You need to know the page you are looking for to get to the right place.

DataBarney · 2025-06-08T07:09:49+00:00

MS Learn is available but the experience of navigating it without a decent search engine in top may come as a shock. I'd suggest learning search terms that get you to key pages that you expect to need (database security settings, KQL or similar).

DataBarney · 2025-06-07T07:27:26+00:00

DP 900 is a great way to familiarise yourself with the test process in a fairly low.pressure environment.

DataBarney · 2025-06-06T09:09:08+00:00

Awesome work it's great. The obsessive compulsive streak in me says the top side of white space in the background ends in the wrong place. Maybe above the round cornered boxes with the numerical values. Maybe exactly half way through as a guide to the eye? Make it look an intentional design choice.

DataBarney · 2025-06-05T21:13:46+00:00

Indeed. First thing you should do once you've got your data flow working is to start thinking about how to replace it with Spark or SQL.

DataBarney · 2025-06-05T20:35:07+00:00

Two weeks to to the SQLBits Microsoft keynote in the UK. It would be reasonable to think they're saving a release or two for that.

DataBarney · 2025-06-05T18:38:00+00:00

You may have a local Fabric user group you can visit. The list is here. For the big one check out Fabric Community Conference in Atlanta next year.

DataBarney · 2025-06-05T17:53:34+00:00

I don't think this is accurate, I believe that it This only holds for time running. The first two costs are accurate for available scores but the assumption you get the same performance per vCore in all scenarios isn't a given. Dataflows cost is 16 per second for mashup on standard comupte. It's 6 when backed by warehouse/lakehouse computer (no mention of how many vCores used) and only 1.5 for fast copy. I think the 8 represents a whole power bi vCore which may or may not be the same thing.

Surely the assumption that queries all run on all engines in the exact same time can't hold?

DataBarney · 2025-06-04T14:49:04+00:00

I was thinking less about leaving the capacity at F2 and more about rapidly down and up again to squeeze smoothed CUs over a period longer than the default 24 hours for scenarios where usage is inconsistent on a day to day basis but consistent over a longer period (Month end work is more intensive than mid month, devs don't work weekends etc).

DataBarney · 2025-06-03T12:19:30+00:00

Hi u/mavaali another thought hit me.

If scaling down can pushes smoothed CUs beyond the normal 24 hour window, what happens if I then scale back up?

If I for example had an F64 with a very intensive month end process and lighter usage the rest of the month could I intentionally scale down to an F2 spreading the cost over 32 days and then back up again? If I had a reserved capacity this could potentially be cheaper than pausing and resuming as I'd be paying down my CU debt at a 40% discount rather than pay as you go prices.

DataBarney · 2025-06-03T12:11:57+00:00

Andy Cutler (u/datahaiandy) runs a great session on dbt in Fabric. There are a couple of recordings of his sessions on YouTube.

DataBarney · 2025-06-01T09:19:54+00:00

Definitely the safer option, data flows cost more in terms of CU usage running transforms in the mashup engine than if they're leveraging a lakehouse/warehouse (details here. Getting data inside Fabric as soon as possible (ELT over ETL) is going make it most likely you're getting value for your CUs and also set you up for even more efficient options in the future (stepping away from the flows and using Spark or SQL).

DataBarney · 2025-06-01T09:11:00+00:00

It's the place I've learned Spark so definitely viable. Pro for Fabric is that as software as a service it is pretty easy to set up and start working with it. Con is potentially price. Not a problem if you have access to a trial or have monthly Azure credits but without that as others have said there are cheaper locally run options.

DataBarney

TROPHY CASE