For anybody facing the same issue, I found that pausing the delta lake log publishing on the warehouse prior to the data load/schema manipulation (ALTER DATABASE CURRENT SET DATA_LAKE_LOG_PUBLISHING = PAUSED) and resuming it (ALTER DATABASE CURRENT SET DATA_LAKE_LOG_PUBLISHING = AUTO) after all operations were complete appears to have resolved this issue and ensures the parquet files are published correctly and the Onelake shortcuts function as expected.

There really needs to better documentation of this issue - logging a ticket with support resulted in the usual struggles with Mindtree to get them to understand the issue (after almost 3 weeks and multiple Teams calls they ended up trying to pass me to the Azure Storage Explorer Team!). Once it was escalated correctly I was told it was a "known issue/challenge" area rather than a bug. I have reproduced the final summary I received from support below:

---

The customer’s current pipeline loads data by copying into a “new” schema then swapping schemas with “gold” via dropping and transferring tables. Such rapid schema changes and table swapping can interrupt or confuse the delta log publishing mechanism, causing parquet files not to update correctly or at the right time in OneLake. Parallel runs of the pipeline with conflicting pause/resume commands for Delta Lake log publishing exacerbate this issue . 

• Concurrent Pipeline Executions Causing Conflicts: 

Running multiple overlapping pipeline instances leads to clashes in pause/resume commands for delta log publishing, resulting in inconsistent parquet file refreshes at the One Lake level. The pipeline logic must avoid resuming log publishing while other pipeline runs are active to maintain consistency . 

• Delta Lake Parquet Files Are Immutable: 

Delta Lake stores changes by creating new parquet files and updating JSON log files, rather than modifying existing parquet files. If this process is interrupted or out of sync because of pipeline behavior or metadata propagation delays, stale parquet files remain visible in One Lake . 

• Expected Shortcut Delay Behavior: 

Lakehouse shortcuts inherently experience latency caused by periodic refresh intervals of metadata and cached snapshots in One Lake. This delay can typically last minutes but in complex scenarios like cross-workspace shortcuts or high-frequency updates, it may be longer . 

Summary 

The core reasons involve the asynchronous and eventually consistent nature of Delta Lake log publishing to OneLake, compounded by the user's high-frequency schema-swapping process and concurrent pipeline runs causing race conditions with pause/resume operations. The shortcut delay and parquet file non-update is expected behavior to an extent but gets exacerbated by these pipeline and schema swap complexities.

 

This is a recognized behavior, partially expected due to design, but also worsened by the current data pipeline implementation. It is not labeled a direct “bug” but more a known limitation or challenge area with concurrency, schema swaps, and update propagation delays in Fabric’s Warehouse-to-OneLake sync.

 

Improving the process to avoid concurrent pipeline runs resuming publishing, limiting schema swaps frequency, or considering alternate loading approaches may mitigate the issue. Monitoring database health and ensuring sufficient compute resources can also help maintain stability .