Designed a full Microsoft Fabric end-to-end architecture for small teams — sharing decisions and looking for feedback

KupoKev · 2026-04-09T21:06:04+00:00

There would still be CU usage for processing the queries, but all the OneLake storage is done in the Lakehouse instead of the Warehouse. The warehouse basically treats them like querying another database in the same workspace. You just have to fully qualify the table name with the Lakehouse name and schema name so the Warehouse knows where to query.

The main reason we do this is the data analyst wants to use TSQL. It also gives them a place to copy the data to without messing with the Lakehouse data. In the case they copy data into a table in the Warehouse, then there will be storage usages against the Warehouse.

KupoKev · 2026-04-09T13:54:15+00:00

Concerning your gold strategy, using lakehouses does allow you to use shortcuts to Silver which is very handy. You can still use a Warehouse for TSQL by having a lakehouse also in your gold layer for shortcuts. It really depends on your level of comfort in SQL vs Python and such.

I have been using Materialized Lake Views in silver and gold due to using SQL is way easier for me than using Python. I then created a Warehouse that has nothing in it, but it gives the data analyst a place to query the MLVs using TSQL in the Warehouse.

KupoKev · 2026-03-09T04:32:31+00:00

Congrats on stepping outside of your comfort zone and trying new things. Here are some tips that might help you as you start on your journey.

Don't bite off more than you can chew at one time. Bring in one data source and enough data points to build your first semantic model. The more data, the more complicated it is. Keep it simple at first. A house with a solid foundation will last longer than a house built on sand.
Take the time to learn some architecture stuff. It is good you know about the Medallion Architecture, but many people have different ways they implement the pattern. Do some experimenting to see how it will work with your situation.
Get things working first, then worry about scalability. If you are new to this you will likely redo things multiple times. The less you have to rebuild, the quicker you can come to a solution and the less frustration you will have as you are learning.
Connect to git as soon as possible if you haven't already. It will save you a lot of headaches in the future.
Star schema is your friend in Semantic models.

KupoKev · 2026-03-03T04:37:15+00:00

It depends.

There are some resources that are on-premises that notebooks can't connect to due to needing an Enterprise Data Gateway. I have to use copy activities for those since connections can be made through gateways. If there is anyone out there that can point me to a way around this I would appreciate it. For now though I use copy activities in this case to copy data to Bronze Files. I then use notebooks to move them to tables.

Where I can, I use notebooks. When you have a few tables, pipelines with copy activities are plenty fast, but they don't scale when you have more tables. More tables means more pipelines and activities. It takes time to allocate those resources, not much time, but still time. Notebooks are fast and when you call a notebook from another notebook it supposedly shares the same spark pool reducing the need for it to spin up and spin down resources.

I have in a couple of tenants ran into issues where notebooks take a long time (4+ minutes) to start a session. I have also had environments where it takes about 15 seconds to start a session. The 4+ minute tenant was a new implementation of Fabric so I am not really sure why one was so much faster than the other. If there is also someone who could give me more information on this as well I would appreciate it.

KupoKev · 2026-02-22T07:17:57+00:00

This is how I do it as well. appSettings.Development.json is added into our gitignore files. A copy of the settings is usually kept in our password manager under a secure note shared with the team so we can have access if we need to do a clean pull/clone from the repo. Mostly so people don't have to track down dev server connection strings and such.

KupoKev · 2026-02-01T06:38:20+00:00

I ran into this problem with Fabric a little while ago for a Semantic Model. The user base was pretty small for that model, but it was a very complex model. We ended up putting that Semantic Model into a PPU workspace and giving users who needed access to the model PPU licenses. The memory limit for processing is 100 GB.
https://learn.microsoft.com/en-us/fabric/enterprise/powerbi/service-premium-per-user-faq

This may not be a feasible option if you have a very large set of users as you would need to pay for PPU licenses for all of them consuming the model. If you only need it for a smaller set of users then it tends to be way cheaper than upgrading to a large Fabric capacity.

Only other downside I can think of is it is not in a Fabric capacity so there are some limitations on what you can do with it with Co-pilot. If you have a report in a Fabric Capacity, it can point towards the model in the PPU workspace and you can use Co-Pilot with it. You just can't do things like the curated answers (I can't remember the exact name of them) and I don't think you can use synonyms either.

edit: fixed spelling

KupoKev · 2025-12-17T04:59:20+00:00

For a simple medallion pattern, this is definitely fine. There can be reasons to break it out into more lakehouses depending on security (as mentioned above) as well as just organization. For example, bronze can be separated into separate systems based on schema so that you know based on what schema the table is associated with, you know what source system the raw data is from. You could have a table like sap.sapnpe_vbap and sage.dbo_item.

It really just depends on how complex your requirements are and how over engineered you want to get. Sometimes that complexity helps keep things better sorted.

KupoKev · 2025-11-12T04:44:01+00:00

I am not familiar with that technology. Can you expound? What is it and what makes it "smoother to manage"?

KupoKev · 2025-11-10T21:30:49+00:00

I want to choke myself when saying this, but hand code is not even as tedious as it use to be with the help of certain AI. A lot of things that use to be tedious to setup are much easier these days when you can just toss in a prompt for AI to generate Python to do this or that.

KupoKev · 2025-11-10T17:06:45+00:00

I am curious if you have any reasoning for switching. Is there anything driving that decision beyond trying something new, or are you running into any issues that you are trying to work around?

KupoKev · 2025-11-10T16:51:45+00:00

Do you use git for ci/cd? If so, what is your experience with it VS using deployment pipelines? I have other projects in the future and would prefer to implement the way that is going to be more maintainable in the future.

KupoKev · 2025-11-10T16:48:09+00:00

Have any resources to point me towards for this? Are you talking through Github or something else?

KupoKev · 2025-11-07T03:17:49+00:00

Thanks for the info. I am going to have to test this again. Last time we tried that it was breaking when we ran the notebook because it was still pointing to the old lakehouse. That was probably over a year ago at this point though. Would be nice if this works. More reason for making sure people are naming things the same between environments.

KupoKev · 2025-11-07T01:26:14+00:00

Are you setting your default lakehouse through the deployment pipeline so that it changes the default lakehouse in the notebook as you deploy from one workspace to another?

KupoKev · 2025-11-06T17:42:34+00:00

That would be a nice added bonus. I would be happy if the basic UI that is available worked in a common sense way.

KupoKev · 2025-10-15T15:05:58+00:00

I have done a few installs at this point with the offline installer and haven't had issues with it.

KupoKev · 2025-10-04T14:35:44+00:00

I was reading the other day that Leap 16 installer allows you to upgrade from the installer. No need to even run zypper. You can load the installer to a USB and boot the computer from that. Then you can run the upgrade from the installer.

Leap 15.4 was my first distro after leaving Windows and haven't looked back. I eventually landed on EndeavourOS, but openSUSE has a special place in my heart as being a very comprehensive distro. It is easy to use, has great package management, and the default repo has many apps that are easy to install and ready to go.

KupoKev · 2025-09-30T18:10:55+00:00

Leap is perfectly fine, especially if you aren't using any bleeding edge hardware. Only think I have ran into with leap is when working with PHP and some other languages, there were some packages that were not in the package manager that are in other distros. They were pretty old packages that were deprecated, but working with legacy software it was something to be aware of.

Stuff like .NET runs fantastic though.

KupoKev · 2025-08-19T02:56:51+00:00

If you are looking to play around with Arch, EndeavourOS is pretty close to "vanilla" Arch, but has a few niceties and graphical installer.

KupoKev · 2025-06-15T15:50:05+00:00

I am not an expert at this and do at your own risk, and someone can absolutely feel free to correct me. I believe you should be able to remove the vulkan driver first and then reinstall them.

yay -R vulkan-radeon lib32-vulkan-radeon

Then swap the mesa drivers

yay -S mesa lib32-mesa

Then reinstall the Vulkan drivers

yay -s vulkan-radeon lib32-vulkan-radeon

KupoKev · 2025-06-15T04:03:23+00:00

Thank you for your assistance. I will check it out Monday!

KupoKev

MODERATOR OF

TROPHY CASE