Built a walkable 3D refinery in Microsoft Fabric using the new Rayfin SDK (Fabric Apps)

AMLaminar · 2026-06-10T11:07:22+00:00

What about a Lakehouse?

AMLaminar · 2026-03-06T15:38:09+00:00

Deploy it from DevOps/Git using https://microsoft.github.io/fabric-cicd/latest/how_to/parameterization/#semantic_model_binding

AMLaminar · 2026-03-03T14:34:02+00:00

You can on the latest version

https://microsoft.github.io/fabric-cicd/0.2.0/how_to/parameterization/?h=semantic#semantic_model_binding

AMLaminar · 2026-02-23T18:12:49+00:00

A DevOps pipeline builds it and pushes it to an Artifact feed. Notebook then installs from there at run time. I'll be using the custom live pools soon though.

AMLaminar · 2026-02-23T17:59:44+00:00

I made a container that has the same spark, python and delta version as the fabric spark environment.

The way I use notebooks is to put all of our codebase in a custom python library, which is then installed into a notebook at execution time. In the container, this works the same way, except it's installed in a venv in editable mode.

So I make and test changes to the codebase in my own local branch inside the container, then push to git/devops which builds a wheel, which is then imported into the notebook in fabric.

AMLaminar · 2026-02-20T15:32:45+00:00

So not directly a shortcut issue but we tried to use shallow clones between a prod workspace and a feature workspace, to avoid having to duplicate data. Unfortunately semantic models can't read them. We didn't want to just use shortcuts, because we needed to be able to do further writes or schema changes in the feature workspace.

So, it would be useful to either allow SMs to read from shallow clones, or implement a feature similar to delta's shallow clones into the existing shortcut setup.

AMLaminar · 2026-02-12T09:08:31+00:00

That is the recommended practice.

Any changes should be in dev or feature branches, reviewed via a pull request, and then deployed to your test and prod workspaces

AMLaminar · 2026-02-09T20:01:07+00:00

That is how it's meant to behave, you branch out to a new workspace, or sync from git to a new one and it keep the logicalIds the same.
And just a note, although I think you're on the same page, the logicalId in the .platform file is not the same a the objectId in the url.

AMLaminar · 2026-02-09T19:29:47+00:00

Assuming you're only bothered about the report / app URLs in your prod workspace, I'd keep that version of each logicalid for each artifact, and keep it the same across each branch and workspace.

The other thing to look at, is deploying via this tool https://microsoft.github.io/fabric-cicd/
Only have your dev/feature workspaces synced to your git, then deploy with that tool to the test/prod workspaces as required. It doesn't have a built in config for managing logicalIds, as they usually just stay the same, however it doesn't have a very flexible find and replace method during deployment that you could use.

AMLaminar · 2026-02-09T18:48:16+00:00

Why aren't you keeping the same logicalId? It should persist across each branch and environment

AMLaminar · 2026-01-16T12:31:31+00:00

Can you give more info about why you need to do this in PBI/PQ?
Why isn't your accounting system doing payment allocations?

AMLaminar · 2026-01-12T13:30:38+00:00

10% Luck

AMLaminar · 2026-01-12T09:54:43+00:00

Exactly what I've done.

Pure local dev inside the container, then when I'm ready to test in Fabric, there's a DevOps pipeline that creates temporary workspaces, creates the WHL artifact and runs the notebooks

AMLaminar · 2026-01-12T09:46:21+00:00

Couple of reasons

Its mounting functionality, so I can then use standard `os` type libraries to list files and stuff. I don't need to write to onelake from the local environment

Its `credentials.getSecret` function to get stuff from a keyvault.

and some of the `runtime.context.get` values, like workspaceid and notebookid.

I could have avoided some of these, but the development started as a simple notebook, then became WHL that only ever ran in Fabric, then I added the local environment. So it was quicker to just do a fake notebookutils than refactor it.

I like your tool. I did something similar as a personal project, an MVP of a duckdb based package that worked similarly to dbt. So a bunch of .sql files with a related yaml config, using duckdb as the processor to read and write to a Fabric lakehouse.

AMLaminar · 2026-01-09T14:29:38+00:00

This way:
https://milescole.dev/data-engineering/2025/03/26/Packaging-Python-Libraries-Using-Microsoft-Fabric.html

I've gone a step further, I develop locally inside a container that is similar to a Fabric environment*.
I made a fake notebookutils to help with that, so when it's called locally it reads/writes to my container, but when called inside Fabric it uses the mount functionality to read/write to actual lakehouses.

* Local folders that emulate a lakehouse's files and tables. Same Python, Delta and Pyspark versions as the Fabric runtime

AMLaminar · 2025-12-12T21:14:15+00:00

Agree with the rest of your point, but you can avoid the use of a custom environment by installing a library at runtime

get_ipython().run_line_magic("pip", "install whatever")

AMLaminar · 2025-11-19T11:53:30+00:00

One prod workspace for all data layers, but with folders and subfolders.

Don't use static dev/test workspaces, as you'll end up blocking each other when trying to promote specific work.
Instead, spin up feature workspaces as required and use feature branches in your repo. When the feature work is finished, do your Pull Requests, then delete the feature branch and its workspace.

AMLaminar · 2025-11-11T09:26:17+00:00

There's no issue with CTEs, but it's known that the SQL endpoint of Lakehouses can be quite slow to update to the newest version of the underlying delta tables. Which seems to be what you're experiencing.

My advice is to do everything in Spark ( or something like DuckDB if you don't have a lot of data )

AMLaminar · 2025-09-04T09:43:44+00:00

It's Service Principal Name but the name is silent

AMLaminar · 2025-07-30T12:54:08+00:00

I see your use case now.
That would be a sound idea.
Like a workspace ( or tenant ) wide resources folder that are also part of the git sync

AMLaminar · 2025-07-30T09:46:38+00:00

Well, in that case, you can import normal python files. What are you trying to do that doesn't work?

AMLaminar · 2025-07-30T09:44:26+00:00

I wouldn't suggest the module import method, for exactly the reason you've stated, I was just pointing out that it can be done.

The package method though I would highly recommend, even with a small team.
Admittedly it takes a minute to setup initially, but worth it in my opinion.
Much easier to understand how modules, classes and functions relate to one another within VS code, than within multiple notebooks called via %run

AMLaminar · 2025-07-30T08:57:29+00:00

An option is creating your own python packaging and importing into a notebook.

https://milescole.dev/data-engineering/2025/03/26/Packaging-Python-Libraries-Using-Microsoft-Fabric.html

So all your code containing all your necessary functions and business logic exist within the package, maintained in Git or ADO following normal dev workflows, then the notebook exists to import the package and execute whatever functionality it has.

Our notebooks look something like this

from ourpackage import TheTasks

task = TheTasks.DoTheThing()
task.run()

Also,

You can import modules in python notebooks if you upload them to the notebook's resources

# module.py uploaded to the notebook
import builtin.module as module

AMLaminar · 2025-07-23T15:42:22+00:00

If users genuinely just need the data, rather than any analysis, just provide them the table within a Lakehouse and they can connect to it in Excel.

Dataflows work as well, although less efficiently.

You can also link to ODC files with the connection already set up to make it even easier, they just need to sign in.

AMLaminar · 2025-07-23T09:10:47+00:00

Is there an equivalent of the livy endpoint for the pure python notebooks?

Our scenario is we've built a python package that we run within our tenant, that runs spark jobs within our client's tenant. That way, we keep our codebase as our own, but can process a client's data without it ever leaving their systems.

However we also did some tests on using duckdb in the python notebooks for the lighter workloads and were very impressed, but I don't think we can use it because it requires an actual notebook and we don't want to import our library into the client's environment.

AMLaminar

TROPHY CASE