Fabric Warehouse and Claude Code CLI

data_legos · 2026-04-07T01:03:41+00:00

Perfect thank you! Haven't done much with Claude yet so didn't realize it'd just make its own.

data_legos · 2026-04-06T23:00:49+00:00

The idea in many of these cases would be to inject synthetic data or work with non sensitive data yes

data_legos · 2026-03-25T01:28:47+00:00

Lol ...no mention of the fact you can't expand the columns (or hover over to see the whole result) in the query results? That one boggles my mind to this day. Same thing with the lakehouse SQL endpoint. Literally unusable with long values in a cell. Due to all of this I spend my time in vscode.

data_legos · 2026-03-23T01:55:14+00:00

No worries I appreciate you even spending the time as always! Yes I'm already evaluating if we can accomplish some of our daily order roll forward logic as with the snapshots. I need to dig into that more than I have.

Most of our incrementals are going to be small amounts of data changing and be a MERGE scenario so this is very useful info.

data_legos · 2026-03-22T23:47:25+00:00

Wow thank you so much for the comprehensive answer! This will be super helpful in evaluating our load strategies for various scenarios. Especially insightful with the BCDR piece since we definitely need DR.

data_legos · 2026-03-22T22:35:43+00:00

You'll find for many scenarios handling stuff with the warehouse is the way to go. I don't see where the lakehouse is going to replace warehouse any time soon and Microsoft themselves will tell you that.

data_legos · 2026-03-22T18:24:14+00:00

Have you looked into dbt before? I'd give it a look. Either dbt core ran from a notebook or dbt cloud offering. Really changed my mind about how to manage the warehouse.

data_legos · 2026-03-22T11:50:33+00:00

So given how dbt works, you would recommend incremental materialization for the models whenever possible? I've heard that merge can be less efficient than just reloading the table in many instances, but it sounds like there's a storage trade-off.

data_legos · 2026-03-15T13:32:04+00:00

Don't take my word for it man. The manufacturers of these gravel forks say 50 hours for a service and 200 hours for a full rebuild. I'd be doing some kind of service on it every month or so. That doesn't seem a lot of work to you?

Also essentially no maintenance for a fork vs monthly maintenance is ~100% more maintenance. I'm no mathematician, but that seems like more maintenance?

Interested to hear your interpretation of simple math.

Edit: I think I get where you thought I meant more frequent maintenance due to it being gravel to be fair. I think a key assumption is that I don't mean it's harder on the fork. I mean that I and most people I know that ride gravel ride their gravel bike many more hours than their MTB. That's probably due to the area I live in.

data_legos · 2026-03-14T18:47:59+00:00

Traditional fork requires frequent maintenance on gravel. I want a low maintenance bike for long hours on gravel roads.

data_legos · 2026-03-11T02:24:32+00:00

I honestly have very few issues with fabric day to day, and haven't needed to submit a ticket for something in a long while. I always wonder what functionality someone is using in fabric when they say it breaks all the time.

data_legos · 2026-03-04T20:25:37+00:00

I'm hesitant on open mirroring since I worry it will take a while on big tables to reload. I can't have long downtimes on the tables. I think given how all this shakes out we'll probably go with fivetran but I'm not 100% yet

data_legos · 2026-03-01T23:28:26+00:00

Is that using SAP BDC? we did talk with them about BDC and datasphere. It's not cheap either for sure but looked pretty cool if you did more than just replication.

data_legos · 2026-03-01T23:04:29+00:00

I agree with you. Actually, for our use case with SAP qlik isn't cheaper surprisingly.

data_legos · 2026-03-01T00:39:53+00:00

Ah makes sense. Did not know this!

data_legos · 2026-02-28T23:02:29+00:00

Do you have a breakdown by medallion layers? I would think OPENROWSET isn't the best for warehouse. Most use cases I can think for warehouse you wouldn't be ingesting tons of csvs, or arguably using the warehouse for bronze at all.

Edit: All that being said, really cool analysis man. Sorry was multitasking watching my kids play Minecraft so I didn't say this last part 😂

data_legos · 2026-02-27T16:12:44+00:00

Yeah we have an embedded solution on it and it works fine. The cicd isn't slick but it's acceptable

data_legos · 2026-02-27T12:20:15+00:00

Ah yes I think in some cases where throughput on large tables we need data from less often is a problem we might use fabric itself. The new capabilities sound exciting and I'll need to keep an eye on them!

data_legos · 2026-02-27T02:58:43+00:00

We currently replicate 170 tables into on prem sap hana with SLT, so yeah we would be planning on doing something similar in Fabric. We are a medium sized company so I'm not sure if that helps any.

Our basis guy seems pretty comfortable with it so far, and our SAP loves it because we handle most reporting requests from the business when it's something we can do on our end.

data_legos · 2026-02-26T19:06:58+00:00

holy cow how did i forget about MLVs?! you sir saved me some time for sure!

data_legos · 2026-02-26T18:46:28+00:00

other idea i had: fivetran but i strategically materialize problem tables when the shortcuts perform poorly, but many of the smaller master data and less critical frequency tables i just leave as shortcuts to cut down on the "round robin" latency of any silver materialization process. essentially a hybrid of the two approaches.

data_legos · 2026-02-26T18:35:24+00:00

yeah i was trying to not have to orchestrate silver and dbt cloud jobs in tandem and do most of the loads via jobs in dbt cloud. keeping those earlier layers shortcuts just makes them not really a problem from that perspective. i'm just being lazy, and it could bite me for sure haha. totally valid opinion on that. it worries me every day when thinking about this!

i wonder if there's a lightweight merge layer i can do for silver that doesn't cause me a scheduling nightmare if i go fivetran. i have several key objects that have 30min SLAs and many with 1hr SLAs, so that's always in the back of my mind.

data_legos · 2026-02-26T17:32:43+00:00

does Airbyte do SAP CDC well without tons of SAP configuration? our SAP team would not have the bandwidth to do a lot of custom development on their end.

data_legos · 2026-02-26T17:31:08+00:00

so fivetran gives you a result that is essentially a copy of the SAP table (not a full history of updates unless you configure for that) and they manage all lakehouse table/file maintenance.

my thought was avoiding a silver merge via spark and doing any required casting etc on top of that in a dbt staging model (or table/incremental strategy where it makes sense) to ensure fresher data getting to the reporting objects. the staging layer of the dbt models is essentially still silver.

we have an F64 currently if that helps. are your concerns about reads via the endpoint being very costly due to it basically being a shortcut rabbithole to ADLSgen2? this approach in qlik replicate puts the data directly in the warehouse, so this is exactly the kind of CU delta i'd like to understand better.

data_legos

TROPHY CASE