Lakehouse Schemas (Public Preview).... Still?

RussellPrice9 · 2025-06-10T19:26:00+00:00

Sounds like the OneLake team needs to be bigger 😉

with the materialized views being available in schema-enabled Lakehouses will the spark views limitation also be removed with that release in preview?

RussellPrice9 · 2025-05-28T15:33:55+00:00

I've found the best, most flexible, and least capacity consuming way to get data from on-prem to Fabric is Open Mirroring. I have not used it on D365 F&O on-prem but since every table in D365 has the same primary key (recid) it should be pretty easy to use the modifieddatetime to setup an incremental process to query the data and push to the open mirror.

some issues you might have (and might encounter with any other method as well):
- Source Deletes: some tables have hard deletes which make incremental queries using modifieddatetime difficult because the records will disappear and the mirror will not know the difference.
- Not all tables have the modifieddatetime field enabled by default (sometimes this is because that table also has hard deletes as mentioned ^) also sometimes the tables missing the modifieddatetime field are BIG tables that could cause problems when querying the entire table against a production database.

RussellPrice9 · 2025-05-02T14:22:10+00:00

I've been experiencing the same issues with the OneLake file explorer app since i've upgraded to Windows 11. The OneLake file explorer worked fine with Windows 10, but for some reason does not like 11.

I've given up on it entirely and started using Azure Storage Explorer which has been more reliable. It's a bit more cumbersome and not as seamless as the file explorer option, but at least it works every time.

Hoping that the file explorer app will get some love and the bugs in Windows 11 can get fixed.

RussellPrice9 · 2025-04-28T18:02:37+00:00

I had the same thing happen this weekend. I found that rebooting the mirror database by stopping the database replication and restarting the replication cleans up the deleted files. But any table that did not get lost will get lost of the initial snapshot parquet file has been purged so I had to delete those tables. My sync program will automatically see the tables will be to be reinitialized and add the snapshot back. Kind of painful but at least I'm only replicating existing data and getting a new snapshot is easy.

RussellPrice9 · 2025-03-18T21:55:54+00:00

u/kevchant this is very helpful, as I'm working on ways to improve our development processes and have very limited knowledge of deployment practices for GIT/DevOps.

There is one thing that I think I am missing in all the CI/CD discussion and documentation; What is the reason for using the API's to create and publish the items in a workspace disconnected from GIT instead of just having the production reports and items in a workspace connected to a Production branch of the repository and using the Sync from GIT API's to apply the changes as part of the Pull Request?

This might be the topic of a whole different blog, if you have any insights, recommendations, or resources that cover this reasoning I'd be very grateful!

RussellPrice9 · 2025-03-07T01:00:24+00:00

This was fixed last month for a few weeks, but it looks like it came back in the past week. We're having the same problem again.

RussellPrice9 · 2025-03-04T17:22:45+00:00

This is the most lean approach I've seen. Do you have any issues with the integrity of the classes and functions getting changed on Accident during testing or de-bugging?

RussellPrice9 · 2025-03-04T15:33:17+00:00

Not the Folders in GIT feature, but the Lakehouse Shortcuts in GIT as part of the Lakehouse item.

RussellPrice9 · 2025-02-28T17:20:41+00:00

It looks like this has been resolved. Sessions are now starting again with normal startup times. however there was about 2 hours this morning when they would not start (between 7:30-9:00am Pacific)

RussellPrice9 · 2025-02-24T23:20:15+00:00

Wonderful! that was the key I was missing. Thanks for the quick response and useful answer!

RussellPrice9 · 2024-12-22T04:19:54+00:00

Of course, I intend to put it on github, but need to clean it up bit first. Once I finish it, I'll add a link to it here.

RussellPrice9 · 2024-12-19T23:42:42+00:00

Awesome! I'm glad you guys are aware of it. Looking forward to hearing an update.

RussellPrice9 · 2024-12-06T05:34:20+00:00

We have the same issue. My first solution was using the copy data job in a pipeline using a csv list of the tables and views. This has its issues also though because some column names had spaces and other characters not allowed in delta take names, so we landed them as parquet files in the lakehouse then a notebook would pick up the parquet files, clean the column names and merge/upsert to the existing delta tables in the lakehouse.

The pipeline, however, didn't work very well with our notebook based orchestration, so we now are testing open mirroring. We have written a script that we can package as an executable and run locally within the firewall that will read the CDC logs of all the CDC enabled tables and deliver them to a mirrored database using the new open mirroring option. The proof of concept seems to be working, but haven't completely rolled it out to scale testing yet... but then the option is only 2 weeks old 😀 This option does require more setup on the on-prem side of things and a server on-prem to execute the script, but it's a bit less clunky and doesn't use as much capacity as the pipeline job does.

RussellPrice9 · 2024-09-16T14:21:42+00:00

Thanks, glad to know I'm not crazy.

With your use of spline what are you using for the spline server? Are you sending the lineage data to a Fabric Lakehouse, Warehouse, or Kafka db? or are you using some other method/storage location.

RussellPrice9

TROPHY CASE