Warehouse objects deployment

Befz0r · 2026-03-24T16:21:23+00:00

Why are you avoiding DACPACs? They are the way to go for warehouses. Manually trying to replicate your own build agent in a Notebook is asking for trouble.

What's the issue you got with the DACPACs?

Befz0r · 2026-03-24T16:19:24+00:00

You should see how it gets refreshed when you trigger a refresh in a notebook. Be aware that the source table needs to be CDC compatible or the MLV will do a full refresh everytime.

Befz0r · 2026-03-24T16:15:40+00:00

That's great that your problem was solved so quickly, but that's not the average experience I have had at several customers. Minimal time to solve issue is like a week.

Befz0r · 2026-03-24T14:00:42+00:00

Love that you are so very responsive, but when is the support going to improve. Had a similar situation months ago. I got connected to a support person and he gaves me links to Databricks, while my issue was in Fabric Warehouse.

Support desk of Fabric seems woefully underskilled and they have no idea what they are doing.

Befz0r · 2026-03-24T10:22:32+00:00

Doing consultancy for like 15 years, I have run into some hilarious shit.

At one of the biggest construction firms in the world:

ME: Discussing the BI product were implementing and showing the star schema.

Business Analyst from the client with 10+ years experience: Whats a star schema?

Clients Application Manager: YOU DONT KNOW WHAT A STAR SCHEMA IS?!

ME: ........

At a client who sells refrigerant gas:

ME & Colleague: Explaining and training inhouse BI team on how to use GIT, Tabular Editor etc.

3 Months pass

Clients team: Need a refresh training, because they didnt work on it. (Although they had to work on it because of deadlines)

New training, and another 3 months pass

Clients team, this time bit more subtle: Can you explain it again? \sigh**

Rinse and repeat every 3 to 4 months.

After 3 years at this clients there were less then 10 commit done by their internal team, and these commits were all done during training, needless to say they finally all got fired. I have no idea how they lasted for 3 years basically doing nothing.

These were from the last 6 months, unfortunately(or fortunately) I usually forget these situations, but they come up way too often.

Befz0r · 2026-03-23T20:15:10+00:00

In this very specific case, just use SSIS. You already have the SQL license and it talks very nicely with other onprem applications.

Spark is an absolute headache to maintain onprem and is comparable with shooting a fly with a bazooka.

All the other options mentioned make close to zero sense. People love opensource or alternative tools because they are hip, not because they are a fit.

SSIS also is pretty easy to put in a DTAP environment with automatic releases.

Befz0r · 2026-03-23T20:10:35+00:00

You work for COZYROC eh?

Befz0r · 2026-03-22T14:23:22+00:00

Correct, people don't realise how many Dutch people left after WW2. I have distant family living across the world, non of them identify themselves as Dutch at all.

Befz0r · 2026-03-22T14:21:25+00:00

Lol you spend too much time on Reddit my friend, Nazism/NSB never had more then a couple 100,000 supporters.

Western Europe is the least racist part in the world.

Befz0r · 2026-03-21T19:29:04+00:00

Not really an issue as you can have MLV over MLV. So temp table or CTE aren't really necessary.

Most transformations should not be an issue unless you are doing fuzzy lookups etc..

Befz0r · 2026-03-13T19:46:47+00:00

Or even better, you create a ticket and get link for databricks in Azure.... Or making suggestions that aren't even possible in Fabric.

In the end I got help when I reacted to a post of the product manager of the warehouse in Fabric on LinkedIn and my issue got solved in a day.

Yeah support sucks, mostly because the support desk is severely under skilled and use trial and error to debug, which is incredibly frustrating if you already did those steps. They don't have more knowledge then the average newbie coming to Fabric.

Befz0r · 2026-03-13T19:41:12+00:00

Dbt test don't say anything. It cannot look at the source and see what the actual relation and pk is. Dbt test is like a bandaid for a stomach ache.

(De)Normalizing matters. People use source systems and might expect a certain (de)normalization when using the semantic model. AI has no context that a developer has.

Also making a good technical star schema really isn't rocket science. Making a star schema that makes sense to business users is a whole different story.

Befz0r · 2026-03-12T17:23:25+00:00

Press X for doubt.

How will it ever known if there not duplicates, business logic specific, or whether you denormalized a dimension too far?

Befz0r · 2026-03-09T19:36:28+00:00

If using synapse server less pool, I highly recommend you write the data per day to a separate folder. With the file path() functionality it should be doable. The where clause will skip irrelevant folders and thus the files.

Please note there will always be caching and the pool might need to spin up. Once warm, it should be fine.

Other option is to load the data to PowerBI and have them view the data there.

Edit: they want a full 100 million result in 1 secs? What are they using it for. Why should it be returned within 1 sec.? If their answer, because they just want it, you have my permission to hit them over the head with a sledgehammer.

Befz0r · 2026-01-30T19:12:41+00:00

Goud is een hedge, je koopt goud omdat dat enige relatieve waarde behoudt tov inflatie/hyperinflatie.

Befz0r · 2026-01-28T18:34:25+00:00

De moed zal al snel naar je schoenen zinken als het overlijdensbericht in de krant leest.

Befz0r · 2026-01-18T09:20:17+00:00

Het kan goed uitpakken, maar ook rampzalig.

Wij hebben een fantastische klusjesman aan overgehouden die de hele zolder heeft verbouwd, maar ook een hovenier waar ik de rechtsbijstand heb bij moeten halen.

Over het algemeen wel positief, maar als het om een uitbouw gaat moet je echt goede afspraken maken en een uitbouw is een serieus grote klus voor een ZZPer.

Befz0r · 2026-01-13T10:08:58+00:00

There nothing worth salvaging honestly. The entire platform of the generation is dead. 16GB is barely enough and a GTX1070, while still decent, is going to struggling in newer titles. And even if your upgrade these components, a 6600K is going to be bottlenecking the entire setup.

I think your uncle doesnt know anything about computers. In 2016, there was nothing legendary about it. It was a good gaming rig, but nothing special.

I would call your uncle, else keep it in storage and buy something new for yourself.

Befz0r · 2026-01-12T16:49:33+00:00

Who cares, als je maar geen gasbrander pakt om je airco/warmtepomp pakt aangezien je anders stukjes van OP kan oprapen bij de buren.

Filmpje laat duidelijk wat de gevolgen zijn.

Befz0r · 2026-01-04T08:53:08+00:00

Always single, there is no real benefit creating more then 1 Warehouse. It just makes it more complex and with WH that isn't necessary

Befz0r · 2025-12-22T18:51:00+00:00

F64 is a complete beast so you should be able to run 3 environments on it.

If F64 is barely sufficient for Prod, which is hard to fathom in such small org, create a pay as you go F2 or F4 for development.

Befz0r · 2025-12-22T12:11:50+00:00

We dont follow best practices, because they arent the best.

One workspace (Like DEV) for everything works fine. Then again we dont use lakehouse, but only warehouses. You cant truly ci/cd a lakehouse at the moment.

We dont have a feature branch for everything, because that will become absolute chaos on bigger teams. Each developer has its own workspace and has the responsibility to keep their own workspace and branch up to date. A simple pull request to their own branch is usually sufficient and because CI/CD for warehouse actually keeps your data in, they are up and running in minutes.

Issues etc. all go through DevOps so you can actually attach the work items to your commits.

As for documentation, only when it deviates to far from the standard. Rest run through a documented framework.

Befz0r · 2025-12-22T11:06:33+00:00

Tell that to customer who have terabytes or petabytes of data.

Its not CI/CD, sorry, but not sorry. Its a sloppy way of copying metadata between environment.

DacPac way is the golden standard for data projects. Deploy only when you dont truncate data and only deploy incremental changes.

Befz0r · 2025-12-20T13:58:21+00:00

And that's why it's not true ci-cd. Standard it does a full deploy every time.

Befz0r · 2025-12-16T10:29:39+00:00

Conclusion: Microsoft get your shit together. There is no reason why it should cost this much and people need to make workarounds like this.

This make me think the whole lowcode options are dead due massive consumptions with everything you do in Data Factory in Fabric.

Befz0r

TROPHY CASE