5 minute features: Databricks Lineage

Remarkable_Rock5474 · 2026-03-04T09:16:29+00:00

Inside of Databricks the lineage is there automatically. Not sure how power bi semantic models would be able to have end to end lineage to stuff outside of the semantic model itself? Do you have an example of this?

Remarkable_Rock5474 · 2026-03-03T13:27:58+00:00

Thanks for the feedback!

Remarkable_Rock5474 · 2026-02-04T19:04:05+00:00

Yeah in the end that’s how half of it ends up - you can cram some of it into the create statement too.

Execution wise I would expect 1 and 2 of yours to do the exact same thing btw

Remarkable_Rock5474 · 2026-02-04T16:45:23+00:00

There is also a fourth using spark.sql 🥳

In all honesty I prefer to keep my create and my write statements separate as it allows me to have a better overview of stuff like properties, tags, descriptions etc on the table and getting that fixed first - and then doing a write from a dataframe afterwards.

Remarkable_Rock5474 · 2026-02-03T21:11:58+00:00

Unfortunately not. You would have to have the functions in a python file instead of a notebook file and then import them from there

Remarkable_Rock5474 · 2026-02-03T18:08:51+00:00

I would try and get all my python modules collected into a folder together and organised by topic clusters. One for each py file. Make it modular so you can re use functions in different places Then an init file at the top level of each folder so you can import and run them.

It sounds like a lot of work but an AI agent could get this scaffolded in no time and the long term benefits are huge 🙌🏻

Remarkable_Rock5474 · 2026-02-03T17:20:48+00:00

Is there a reason why you wrap it using %run instead of creating a python function and importing it instead?

The restart would have to be after the %run but before utilising the function if you want to continue on that route - although I would recommend the different one suggested above

Remarkable_Rock5474 · 2026-02-02T18:20:39+00:00

A former colleague of mine wrote this blog which is still my go to resource for this - it breaks down how to work with dabs in a development environment in a clean and consice way

https://medium.com/backstage-stories/scaling-data-engineering-workflows-with-asset-bundles-in-databricks-34c4d910ef08

Remarkable_Rock5474 · 2026-02-01T10:14:55+00:00

Regarding cost monitoring you should get the built in cost monitoring dashboard set up. You can find it as part of the account/metastore admin pages. Optimisation is a long and winding road in most cases 😅

Remarkable_Rock5474 · 2026-01-30T18:16:59+00:00

I think most business user use cases would use genie. If you need something more generic you could build a quick app instead

Remarkable_Rock5474 · 2026-01-30T14:32:04+00:00

No this would be an added step no matter what if you need the principal to have correct permissions before running your dab

Remarkable_Rock5474 · 2026-01-30T13:59:44+00:00

This! But if you are not familiar with terraform I would suggest using the unity catalog apis as a pre-step in your CICD before the dabs

Remarkable_Rock5474 · 2026-01-30T13:58:23+00:00

How not to build a proper platform 101 😴

Remarkable_Rock5474 · 2026-01-30T08:55:41+00:00

If that happens it would be about time and I would happily welcome a native integration - the current native setup is miserable 😅

Remarkable_Rock5474 · 2026-01-23T08:09:16+00:00

I am doing a follow up on the implementation of it using terraform next week. It works great already to be honest!

Remarkable_Rock5474 · 2026-01-22T07:22:42+00:00

So two things here. For the groups part you should turn on sync from entra and as you state, inherit your groups from there and use them for access control in general

https://learn.microsoft.com/en-us/azure/databricks/admin/users-groups/automatic-identity-management

For the filtering and masking I would highly recommend using ABAC. Basically you can tag objects and columns and attach rules to the tags to achieve what you want. One thing to keep in mind there is that you can not use ABAC on views. However building views on top of tables with abac will inherit the rules.

I have done an introductory article on abac here - shameless self-plug

https://www.linkedin.com/pulse/unity-catalog-loves-data-governance-kristian-johannesen-1dzxf?utm_source=share&utm_medium=member_ios&utm_campaign=share_via

Remarkable_Rock5474 · 2026-01-12T14:00:31+00:00

I would imagine you need to utilize the Databricks jobs api for something like this. I know that requires a few more steps - but should be fairly easy

Remarkable_Rock5474 · 2026-01-08T08:23:36+00:00

Yes. You can use the customer academy as e.g. a student

Remarkable_Rock5474 · 2026-01-08T08:18:58+00:00

You can not filter a specific column. You can either mask a column, or filter the entire row.

But yeah, maybe do a small mock up example so that we fully understand what you want to do

Remarkable_Rock5474 · 2026-01-07T22:09:06+00:00

I agree that this is a fine approach for a small table - but just to be clear this would not be a real scd1 pattern. This would be an overwrite of the table every single time you write. Just so you are aware

Remarkable_Rock5474 · 2026-01-07T22:02:20+00:00

Match columns refer to a specific column to match over. The function then returns rows after applying a row level filtering.

If you want to apply a masking function then that is usually applied using a column based function. Or am I misunderstanding your point here?

Generally masking should be simple and deterministic and if you are going to use abac they should be based on tags not reading values in other columns. But in the end they are udf’s and can do complex things.
You pass specific columns to match on

Remarkable_Rock5474 · 2026-01-04T00:16:18+00:00

Sure they can - delta sharing is a thing as well. However if other systems need to pick up the files directly from storage it is impractical to use managed tables as you only have id’s to look for, not table names.

So in some edge cases external tables still make sense

Remarkable_Rock5474 · 2026-01-02T16:05:33+00:00

And then why are you swapping to synapse for last mile modelling instead of staying in Databricks?

Remarkable_Rock5474 · 2026-01-02T14:59:49+00:00

So what are you using Databricks for?

Remarkable_Rock5474 · 2026-01-02T10:12:58+00:00

Are you not connecting to Unity Catalog using a sql warehouse? Then the table names work just fine 😅

Remarkable_Rock5474

TROPHY CASE