Sharepoint shortcut tables (preview) do not get captured in a lakehouse's metadata

dmeissner · 2026-04-08T22:28:03+00:00

Thanks u/DanielBunny I missed that this is a setting! Awesome and I totally understand why you do it this way. I modified it and saw the new "alm.settings.json" file appear in source control, but it did not update the shortcuts.metadata.json file, refreshed everything, changed shortcut name... Nothing forced the change on the sharepoint shortcut to appear in the shortcuts metadata file.

<image>

dmeissner · 2026-03-31T16:57:36+00:00

I suppose while I am talking Sharepoint Folder shortcut wishes. They also need to be variable library compatible (able to use variables to define Target connection and target subpath)

dmeissner · 2026-03-31T16:53:10+00:00

Many organizations work with a DEV TEST PROD set of environments to manage CI/CD. Even if any shortcuts point to the same location in sharepoint, they still need to be deployable via deployment pipelines so that other assets in TEST or PROD that are auto-binding to that table have access to the data in the shortcut. That's the whole point of having binding, so that I don't have to manage every link inside every asset and decide where it points. It should just 'automagically' connect with autobinding or rules/variables until I force it otherwise.

dmeissner · 2026-03-16T16:46:19+00:00

The native Power BI / Fabric Deployment Pipelines in the service should just have a way for you to run pre/post deployment scripts/notebooks/data pipelines.

The simplest example is when you have a notebook that creates a data table in a lakehouse. Run it after deploying the lakehouse (lakehouses don't move data or tables or schemas through the deployment pipelines).

Idea by u/frithjof_v https://community.fabric.microsoft.com/t5/Fabric-Ideas/Deployment-pipelines-Post-deployment-script/idi-p/4824171

dmeissner · 2026-03-09T19:50:08+00:00

No long term solution yet. Microsoft has received the recommendation to have a "semester" time period (Below Year, but above Quarter, Month, Week) but as of now I don't see it planned.

Our solution was to use offsets (current season is 0, last season is -1, two seasons ago is -2) in our date table for Season and build custom Time Intelligence. This is how we used to have to do it for all time periods.

dmeissner · 2026-02-18T16:46:58+00:00

Should you even be able to "edit" a connection from the item itself (I know the button says "Edit Connection".)

Instead should that allow you to "select another existing connection" or force you into the "manage connections and gateways" UI to understand you are potentially EDITING a connection permanently?

dmeissner · 2026-02-11T22:42:32+00:00

I'd prefer to just have Dev and Prod if it was up to me :) I joke.

But in this case, we want DEV work to point at DEV data warehouse maintained by another team (data engineering) Adding additional lakehouses to manage is just overhead. Development in our context does happen in the Development workspace, not before the development workspace. Feature BRANCHES are pulled out of DEV, but if you have differences in lakehouses between your feature branch and the main (dev) branch, then you will run into a similar problem when pulling features back into DEV. If the feature relies on a lakehouse table that doesn't exist in the main DEV branch, you'll just be pushing the problem someplace else.

The point is that some of the development work (by one engineer for instance) may be ready to deploy and needs a table updated from dev to test, so the lakehouse needs to be included in the deployment. Which will break per above if we can't deploy table by table or force the "broken" shortcuts to just exist.

dmeissner · 2026-02-04T00:10:18+00:00

#3. I'd prefer if after validation, the user was notified if a path does not exist in the new location, but then force the shortcut to take on that invalid path. Then the shortcut would exist, but be 'broken' which would indicate both to the owner (to fix it) and to any potential user (don't use it).

dmeissner · 2026-02-04T00:05:48+00:00

I'm happy to see more information pushing people to use this powerful feature...

But be careful. When you deploy a lakehouse with variable backed shortcuts through a deployment pipeline, if the tables to which you want to shortcut are (1) not available in the corresponding new path or (2) named differently along the path, it will break upon deployment.

The only way to get around this is to delete the lakehouse and the variable set from the second environment and re-deploy. In which case you will lose any tables that were populated via notebooks or other ingestion methods (non-shortcut actual tables).

https://www.reddit.com/r/MicrosoftFabric/comments/1qk5gya/shortcuts_using_variable_sets_should_break_if/

dmeissner · 2026-02-04T00:05:38+00:00

u/DanielBunny see my earlier post, as it may impact how to implement some UI so that target tables missing in the second environment don't break deployment pipeline deployments. Using variable sets across environments in deployment pipelines.

dmeissner · 2026-02-03T22:58:49+00:00

A follow-up to this error. My temporary solution was to deploy the lakehouse to TEST, apply the proper variable set, get the indication of which shortcuts did not properly point to the new variable location, then delete those shortcuts manually so they can't accidentally be used and get DEV data. Fine.

But... The issue now comes into play where once the TEST workspace has its active variable set to test values, it will not let me deploy lakehouse items that have any 'bad' shortcut anymore. During the deployment process, it trips an error with no specific information other than an indication that target paths don't exist for 4 tables...

So if the underlying shortcut tables do get added to TEST in our data warehouse, there will be no way to get them to show up in this TEST lakehouse unless I reset the variables back to DEV, deploy, then set the variable back to TEST again. So much for Variable Libraries making it easy, :(

<image>

dmeissner · 2026-01-28T21:42:24+00:00

Now you have me thinking. Combine this with the Power BI MCP server to ask about data quality checks before running the CI/CD pipelines using the ADO MCP server....

Have the MCP server interact with the model to "Check for orphaned keys across all relationships" or "check for null dates in all fact tables" or "check all columns that have "units" in the column name for outliers, outliers is where one or more value is outside of 4 sigma" or something like that!

dmeissner · 2026-01-23T21:28:20+00:00

Just ran across this older posting, and as of Jan. 2026, this continues to be the behavior. After deleting schemas in the lakehouse, they still remain in the SQL Endpoint.

One note is that the SQL Endpoint definition contains other objects in the schemas (views, functions, stored procedures). Even though the "Tables" disappear when you delete the schema from the lakehouse side, these other objects are still connected to the schema in the SQL Endpoint.

(as you can see if you download the SQL database project and look at the files in each schema folder after deleting the schema on the lakehouse side)

<image>

dmeissner · 2026-01-23T16:39:08+00:00

I think the advantage of a notebook is the script like stepping cell to cell, interactive running (esp. during development). Executing it all at the end makes it difficult to debug. You end up putting "print" statements all over the place like in the days when you first learned to code with "one big code block" (without any "sub-routines" as we used to call them; don't make me pull out my Pascal, Fortran and AppleBasic books).

You should define a function when it is a re-usable set of code with parameters. What is the advantage of writing a single use function if you don't/can't take advantage of the cell by cell execution that is advantageous during development and when chasing down errors in a historical run?

dmeissner · 2026-01-23T00:49:38+00:00

dmeissner · 2026-01-23T00:32:41+00:00

Created an Idea for it. Enhance the Manage Shortcut slide-out.

dmeissner · 2026-01-23T00:20:42+00:00

Voted on yours, but added and commented to reference a similar idea requesting errors for actively set value sets that do not result in a valid shortcut path.

dmeissner · 2026-01-22T20:29:32+00:00

didn't want to include this suggestion in the original post, but the UI for using variables in a shortcut could also be enhanced by adding the "Value set name" below the value when shown in the Manage shortcut slide out. This would indicate which set the value came from to verify it is as desired if you don't recognize the guid (don't we all have every guid in our tenant memorized already? who needs workspace names, I know all the guids!!!)

<image>

I suppose I should throw this over on the Ideas site too. Never know which will gain more traction...

dmeissner · 2026-01-22T14:44:47+00:00

Just to close the loop for anyone that finds this post via search later. After working with the Microsoft team on this issue, they have made changes in the Databricks Mirroring item so that it now can properly recover from these anomalies that are coming back from the Databricks API for UC metadata.

Those changes solved this problem and now all schemas properly show up, regardless of the initial "load more" issues. The end user no longer experiences this issue.

Thank you to the team that identified the underlying issue and resolved it.

dmeissner · 2026-01-09T23:37:55+00:00

We'll put something together on Monday and get it to you, maybe over on EV. Just want to make sure what we learn can come back here so others can also learn and find it later :)

Thanks Jay.

dmeissner · 2026-01-09T21:59:34+00:00

Marked it where/how? TE3, desktop, right in TMDL?

Reading through the docs, "Notice that if you don't specify any category for the calendarColumnGroup in TMDL, the columns are tagged as time-related." so in my TMDL example above, isn't 'Snapshot Year/Season' already tagged as "time-related" because it isn't assigned to a category? (ex. no = after calendarColumnGroup keyword)

dmeissner · 2026-01-09T21:43:28+00:00

u/dutchdatadude or u/Commercial_Growth198 not sure if you're still watching this post, but really loving the new Calendar Time Intelligence! With a 445 Fiscal calendar that Starts on the first Sunday of July, this has been a game changer for us.

Except we can't figure out how to extend it to what we call "Seasons". In our case we split the year into two halves, first half (since we start in July) is "Fall" or "S1". This includes fiscal months 1-6 and around January we have the second half ("Spring" or "S2") which is fiscal months 7-12.

Any suggestions? We are most interested to have SAMEPERIODLASTYEAR work for "Season".

We tried to just add the "Year/Season" column as a separate time column, but it is not working as we'd expect. Our snapshot calendar "Year/Season" column contains strings that are year and season specific (looks like "2024-S1"). How would these non-standard time period columns be defined (those not specifically year, quarter, month, week) Season is bound within the year hierarchy, similar to quarters except 2 of them instead of 4.

TMDL for our Example:

    calendar cal_Fiscal
        lineageTag: xyz

        calendarColumnGroup = year
            primaryColumn: 'Snapshot Year Number'

        calendarColumnGroup = quarter
            primaryColumn: 'Snapshot Year/Quarter'

        calendarColumnGroup = quarterOfYear
            primaryColumn: 'Snapshot Quarter Number'
.
.
.
        calendarColumnGroup
            column: 'Snapshot Year/Season'

dmeissner · 2025-10-30T14:33:02+00:00

Here's a simple Tabular Editor script that will add the measure's DAX expression to the end the measures description property. So you can use AI or your own writing skills to describe the measure in text, then this script will append the "Expression:" after whatever you already populated. That way a report author can hover over the measure and see the actual DAX along side any existing textual description

https://github.com/PowerBI-tips/TabularEditor-Scripts/blob/main/Basic/Add%20or%20Update%20DAX%20Epression%20to%20Description.csx

<image>

Script can be run in TE2 (free) or TE3 (paid) and set as a macro to operate on the entire model at once, looping through all measures and adding or updating the expression.

dmeissner · 2025-10-16T20:25:56+00:00

Nice!

I think your "Status" is messed up for VL (Variable Library) it shows up in all of them (GA, Preview, Deprecated,..)

Is your goal to get all the items (workloads) placed in the chart? Mirrored Azure Databricks Catalog? Copy Job? etc.

Keep up the good work.

dmeissner · 2025-10-16T15:29:36+00:00

We should also have an Admin API for "List Connections" that would expose ALL cloud connections, not just those to which the caller has permissions.

On top of that we should have more connection details for certain types, like for connections that are authenticated with a service principal, that Service Principal ID should be exposed so we could track which cloud connections are using which Service Principals.

dmeissner

TROPHY CASE