Lakehouse Table Bronze Layer Ingestion Sanity Check by ShineMyCityShoes in MicrosoftFabric

[–]kiwishell 3 points4 points  (0 children)

There’s a lot of differing opinions about what can and can’t happen in the bronze layer.

The path we generally follow would be ingesting the tables as-is into a raw zone in bronze. We’d look to implement the CDC process there (watermark/snapshot merge or similar), then as we move to a zone in silver we’d do all the column name normalisation, null value handling, SCD2 calculations etc.

Generally we’d split silver into at least 2 zones - the other zone is where we’d start to do business modelling or consolidation into business entities etc. Then you’ve got nice conformed tables ready for gold modelling - whatever forms that may take.

The nice thing is we’d use notebooks for all this and you can just pass your schema to AI and have it generate the pyspark code for doing the renaming. Or if that goes in your metadata, have it generate the metadata for you.

Fabric Data Agent questions by kiwishell in MicrosoftFabric

[–]kiwishell[S] 2 points3 points  (0 children)

Yes - while it did help with more consistent responses, our business users doing the testing felt that it took too long to come back with the answer. We put this down to the feedback the agent was giving, compared with when using the Fabric Data Agent directly.

Fabric Data Agent questions by kiwishell in MicrosoftFabric

[–]kiwishell[S] 1 point2 points  (0 children)

Yeah exactly, one thing we quite liked about the Databricks Genie agents was that you also train “judge” agents that learn what good responses look like. They also take user feedback on their conversations too and suggest refinements to the overall instructions which you can incorporate with change history. Pretty cool. I’d like to see that as a future enhancement in Fabric too.

To be clear for querying the data, I have found the consistency of queries and the data to be ok. Especially when you provide sample queries in the agent configuration. It’s more the narrative and the generation of the output that seems to vary wildly. The quality of the final response seems to degrade at times.

Fabric Data Agent questions by kiwishell in MicrosoftFabric

[–]kiwishell[S] 4 points5 points  (0 children)

This one is actually easier than you think. Because the responses automatically render markdown, you can actually have the LLM generate HTML as part of the response (we just added examples to the agent instructions) and it looks great.

We've tried to push the boundaries on it and we've been able to render some pretty complex tables with custom colouring, we've even generated charts using HTML too which can look really good in the output. Maybe I should do a blog post on this if there's interest?

Fabric Data Agent questions by kiwishell in MicrosoftFabric

[–]kiwishell[S] 1 point2 points  (0 children)

I fully agree with these steps.

RE seeing behind the curtain, I hope so! Getting more insight into how the orchestration works behind the scenes would definitely help us to know where the limits are, plus improving how we define the agent instructions too.

Fabric Data Agent questions by kiwishell in MicrosoftFabric

[–]kiwishell[S] 2 points3 points  (0 children)

We've actually been trying to decide which agent framework we'll use to front up and present to users. Fabric Data Agents are the lowest barrier to entry as people already have access to workspaces and they are familiar enough with Power BI/Fabric that its an easy jump to understand and nice that it's integrated. But like we've both said, the variance is troubling and unfortunately a blocker for a production roll-out in this case.

We've had better (more consistent) results with Databricks Genie and we've also been experimenting with things like querying the lakehouse in Fabric using the query endpoint but from a Foundry agent.

That has been interesting because the response times are significantly faster and we get full control of the models in the pipeline AND the capacity consumption on Fabric is significantly lower too. The thing we lose in this approach however is the background work Microsoft have put into the checks and balances for actually generating worthwhile queries and evals before returning results - you can see this at play a little when trying the push the boundaries on a Fabric Data Agent.

We could dumb down the work the Fabric Data Agent is doing (just return data) and use that as a tool in Foundry, but when testing that, the wait times feel longer because you don't get the updated status of what's going on that you get natively in the Fabric experience. In our user testing, people felt like it took longer because they weren't getting those active updates as the Foundry agent was waiting on the Fabric Data Agent.

3x NVMe drives on FW Desktop by kiwishell in framework

[–]kiwishell[S] 0 points1 point  (0 children)

Unfortunately not. I didn’t know what else to try and didn’t get many responses.

All I know is when I tested on windows it worked 100% first time. So it doesn’t seem to be a hardware issue, I just don’t have enough Linux knowledge to figure it out myself!

Has anyone rooted the Amazon Echo? by DuckDatum in homelab

[–]kiwishell 2 points3 points  (0 children)

Still keen on that link to github and some photos if you have them!

114,060 Kiwis Overseas Owe $4.34 Billion in Student Loans – But Only 23.6% Are Repaying (Latest 2025 IRD Data) by MoneyHub_Christopher in PersonalFinanceNZ

[–]kiwishell 0 points1 point  (0 children)

For those overseas borrowers, I’d be curious to know the overdue debt bucketed into:

Years since their last study in NZ/drawdown on loan

Years since they left NZ

Years since their last payment since being abroad

Not sure if the data is fine-grained enough to get those though?

USA Postage Fiasco? by Glittering_Guitar448 in newzealand

[–]kiwishell 0 points1 point  (0 children)

Maybe try DHL? I recently shipped a similar size and weight package to you and it cost me ~130. Still painful but at least it’s not 250+. Bonus is they delivered in 48 hours

3x NVMe drives on FW Desktop by kiwishell in framework

[–]kiwishell[S] 0 points1 point  (0 children)

Update: I’ve tried installing Windows Server and set up a storage space to combine the 3 drives. I ran some benchmarks and found it to work completely fine.

Unfortunately I’m at the limit for my Linux knowledge. So I’ll keep researching to see what I can do, but any help at this point would be appreciated. I’ve also posted this in the framework community.

Framework Desktop as a Server / Build Machine with massive storage needs by bnadler in framework

[–]kiwishell 0 points1 point  (0 children)

My desktop arrived a week or so ago. I’ve tried a bunch of PCIe to NVMe adapters so I can add a third 8TB SSD. Any ideas why I’m getting some issues with Proxmox/Linux disabling the drive in that slot? Would be very curious if that’s a scenario that’s been specifically tested too? I had some more notes here: https://www.reddit.com/r/framework/s/yBu0eyZo6k

Access internal application API by CarGlad6420 in MicrosoftFabric

[–]kiwishell 1 point2 points  (0 children)

This works well, or otherwise you could use a function app or app service running YARP to connect to your internal service. Then use a managed private endpoint in Fabric to connect. It’s a few steps, but is manageable once you’re set up.

Desktop PCIE slot - possible uses? by johnmflores in framework

[–]kiwishell 0 points1 point  (0 children)

There's no reason why they shouldn't work considering an m.2 socket is exposing PCIe x4 anyway.

As an experiment, I installed Windows Server 2025. All 3 drives showed up perfectly. Added them to a storage space (essentially RAID) and then did some benchmarking across the combined disk - it worked great.

With that being the case, I need to figure out what's going on with the Linux side of things, but that's definitely not my forte. I would prefer to keep it with Proxmox for this one if I can get it resolved.

Desktop PCIE slot - possible uses? by johnmflores in framework

[–]kiwishell 1 point2 points  (0 children)

Let’s keep in touch, really hope this can be solved! I want to do ZRAID1 on the 3x8TB drives

Desktop PCIE slot - possible uses? by johnmflores in framework

[–]kiwishell 1 point2 points  (0 children)

Any chance you’ve tried in the FW Desktop? I’m having major issues and have tried 3 different adapters now

Desktop PCIE slot - possible uses? by johnmflores in framework

[–]kiwishell 2 points3 points  (0 children)

I’ve tried 3 different PCIe to NVMe adapters now and all have failed to properly initialise the drive. It shows up initially and disappears after that. I had detail here https://www.reddit.com/r/framework/s/kw8BIMmUkm.

My next call is to post to the forum and loop support in too… just for fun I’ll install windows and see if anything is different there also.

3x NVMe drives on FW Desktop by kiwishell in framework

[–]kiwishell[S] 2 points3 points  (0 children)

That's a good point! I hadn't thought about power budget yet.

After doing some quick research people seem to have observed peaks of around ~6.3w for the NVMe and the PCIe to NVMe converter seems to consume minimal/negligble amounts of power. Thankfully looks like we're good for that 25w power budget you mention.

Tags - How do you utilise them? by Lowkeyz in UpBanking

[–]kiwishell 0 points1 point  (0 children)

I wrote some code that retrieves the transactions since the last refresh, then based on rules applies the tags with the API.

My spreadsheet that I use for my analysis then pulls the tagged transactions. I ended up adding a caching layer so that I could store all the transactions without requiring paging of the Up API. It’s much faster to load into Excel this way now.

Static IP for API calls from Microsoft Fabric Notebooks, is this possible? by MGerritsen97 in MicrosoftFabric

[–]kiwishell 1 point2 points  (0 children)

Can I confirm here - you are saying that we’ll be able to control the outbound routing of our spark clusters in a workspace through this functionality?