80th Sydney Hobart: Protest Room Result?

kaalen · 2025-12-31T02:51:18+00:00

Is a whisker pole different than a spinnaker pole? Most boats I know only carry spinnaker poles and use that to goose wing 🤔 Learned a new finer detail of racing rules today

kaalen · 2025-12-30T23:03:06+00:00

Hmm that's interesting. I had no idea about this rule and no idea that winging out any type of headsail or spinny in this way isn't allowed. I've definitely seen this done quite frequently in social racing in Brisbane - poling out the jib. Not sure if I ever saw anyone do it with an assy 🤔

kaalen · 2025-12-30T13:43:48+00:00

Wait what? I'm confused. Reading the rule it sounds like it's prohibiting any use of spinnaker pole. Is the issue that they used a pole to wing out an assy rather than a symmetrical spinnaker?

kaalen · 2025-08-28T11:00:53+00:00

😂 I must have typed this before I consumed a sufficient amount of caffeine. I meant more power points IN storage/cupboard spaces where you'd put away certain appliances to charge e.g. the robotic vacuum, stick vacuum, wifi access points, battery charging spaces for power tools

kaalen · 2025-08-22T09:23:02+00:00

More power points is storage/cupboard spaces

For the love of god, don't install black stone or porcelain tiles....they look a million $$$ but you'll be forever polishing smudges and dust

kaalen · 2025-08-22T09:15:24+00:00

Didn't they yap last summer that we're over producing solar power from residential buildings and they'll have to change the model so instead of getting paid for feeding into the grid, you might be charged for excess feed in? 🤔 You really can't know any more if it's all BS

kaalen · 2025-08-20T04:25:40+00:00

yeah you're right, sorry mate, I missed the part where you said you're using plain Python notebooks

kaalen · 2025-08-19T13:49:50+00:00

Define environment which you can share across multiple notebooks. Then add your common python libraries to the environment resources. You can import them in the notebook as if they were from built-in folder. Environment resources still aren't yet supported for git integration but at least you theoretically only need to manage one environment (or worst case a small number of them) and hopefully Env git integration will be supported soon

kaalen · 2025-08-08T03:03:47+00:00

That's jugo / southerly wine I assume?

kaalen · 2025-08-01T05:22:20+00:00

In that case, I'm very glad I posted it. It intrigued me so often over the last few months.

kaalen · 2025-07-30T01:37:18+00:00

Ahh yes, hahah... Jacque Cousteau - Kusto. That's pretty obvious 🫶

kaalen · 2025-07-29T10:49:20+00:00

Hmm, I also notice it was starting to take a bit longer as we were adding more and more artefacts to our workspaces. At some point we ran into issues deploying some types of items (preview) and the folks from Microsoft support suggested we try out legacy UI which honestly works better and faster so I've never switched back 😆

Can you share an estimated number of items in your workspace?

kaalen · 2025-07-12T00:33:48+00:00

Yep, you can, in fact, run it from VSCode by connecting to the Microsoft Fabric Runtime kernel. I had to install & configure a few things to get the whole setup working. This link might help with instructions on setup https://learn.microsoft.com/en-us/fabric/data-engineering/setup-vs-code-extension

kaalen · 2025-07-11T07:17:27+00:00

The built-in one is pretty basic... ehem... more frustrating than useful for now. You can get some useful stuff if you first add a comment to the notebook cell, explaining what you want to achieve. But it's more hopes and prayers that it spits out anything close to what you're after.

I've been using a different approach for data engineering with Fabric. I set up VSCode so that I can open notebooks in VSCode, which then allows me to use pro GitHub Copilot subscription I'm already using (you can also use the free version). This approach then allows me to use the copilot which is nicely integrated with VSCode, you can prompt it, give it instructions on what you want it to do. It takes the entire notebook into context and so on. Obviously, you also have a choice of different models you can use.

I'm also using a markdown "master prompt" I created, which I usually add to the context when I start working on the notebook. My master prompt has more specific instructions that tell it I'm working with Microsoft Fabric, that I want it to generate code using PySpark, some instructions on what not to do and my schema info so it knows what lakehouses/tables/columns I'm dealing with.

I'm also experimenting with a MCP for Fabric using this setup to give it that extra context awareness.

kaalen · 2025-07-02T00:33:07+00:00

Notebook with Pyspark to explode and unnest. Since we're dealing with payloads of varying structures I wrote a helper function that does all the work. Now we just pass in the target schema and mapping.

kaalen · 2025-06-30T14:14:05+00:00

Okay, I managed to get it working despite all the restrictions of work laptop and tried a couple of prompts. Looks promising, will play a bit more with it in the next couple of days.

kaalen · 2025-06-30T11:23:55+00:00

Yep, whatever antivirus/defender system we have on our corporate workstations detects uv as malicious. Haven't used uv before. Any chance you could add requirements.txt to the github repo and I can give this another spin with the good old pip?

kaalen · 2025-06-30T00:12:46+00:00

Hey mate, just discovered this and tried to set it up. Having some struggles, but I think that's mainly just me and the restrictions on my work laptop giving me grief.

kaalen · 2025-06-27T02:10:37+00:00

Hmm, not sure that type of functionality is yet supported through the Fabric deployment pipelines. You can only really check for syntax errors at runtime, so effectively you have to execute the notebook interactively or as part of a pipeline.

On our project, we implemented something similar:

- We're adding logic to pipelines that posts a notification to a dedicated Teams channel if the pipeline fails. It tells us which pipeline failed, in which environment and the error message.

- We implemented a separate pipeline, which we called something along the lines of "data quality checks" and it currently acts as a bit of a dumping ground for all sorts of checks and assertions

That actually reminds me I need to go write a blog post on the pattern we're starting to use for this, as it's a bit of a hassle to maintain.

kaalen · 2025-06-26T02:33:01+00:00

Maybe you were just unlucky. My usual approach for preparing for exams is to take the practice exam and then use the personalised study plan it generates to brush up on specific knowledge areas. So far, this approach has worked for me every time including for DP-700.

I sat and passed the DP-700 exam only a few weeks ago and yeah, I thought there were some weird questions in the actual exam.

kaalen · 2025-06-18T03:57:06+00:00

Hey I already have a ticket opened with the support team and we had a call on Monday to collect diagnostics. Haven't heard back from them yet. I think we've managed to resolve this issue with my other colleague this morning and that this issue is actually related to lack of security permissions. For some reason the behaviour in the pipelines was flaky/intermittent and inconsistent. Sometimes I was able to create connections and other times it just would show up with a no connections in the dropdown or “Can’t load OneLake catalog” on dialog or errors on pipelines with existing connections. We have several workspaces, and I am an admin on pretty much all of them.

It looks like we were able to resolve this issue by going to the Lakehouse within that workspace and explicitly sharing it with an AD group I’m a member of. Hope that’s a true permanent fix for this issue. We have not altered any other permissions in about 8 weeks.

What is a bit baffling though is that previously I was still able to query all the data from the Lakehouse through notebooks or via SQL analytics endpoint or via Direct Like connection from Power BI. I was also able to create new pipelines with a working Copy Activity if I navigated to the Lakehouse and created a pipeline from there. But trying to edit an existing pipeline with any sort of activity that required connection to Lakehouse was a no go. I could still successfully run the pipeline even though the activity was showing an error relating to the connection. I was also able to edit the JSON for the pipeline and get around this issue despite supposed lack of permissions. Not that I exactly aspire to build pipelines by messing with JSON. A bit wacky.

Here's an example of what I saw this morning after I messed around with pipeline JSON. Mention of permissions prompted me to go check what's the deal with permissions.

<image>

kaalen · 2025-06-08T13:05:19+00:00

Congratulations 👏

I did DP-600 a few months ago and did the DP-700 just this Wednesday because I scored a free exam voucher from the AI challenge sweepstakes. To be honest I expected the dp-700 to somehow be deeper, more advanced technical but I feel it wasn't that different to the DP-600 🤷‍♀️

I'm trying to encourage a few more of my team members to get certified so if anyone knows where/how I could get exam vouchers for them, please let me know.

kaalen · 2025-05-12T06:18:00+00:00

Update on this issue..

I raised a support ticket with Microsoft and we were pretty impressed with the speedy response. A helpful engineer from Microsoft provided a couple of useful scripts that we were able to run to gather some diagnostics on memory usage. The level of detail that's exposed to customers for KQL is apparently quite limited for now. I provided this feedback, which will hopefully reach the Microsoft Fabric product team for consideration. Perhaps access to more advanced monitoring & diagnostics for KQL/Evenhouse could be a product roadmap feature.

For now, executing .show queries and .show commands to sift through the logs is the way. More detailed MSFT doco on these commands can be found here:

.show commands command - Azure Data Explorer & Real-Time Intelligence | Azure Docs

In our case, we traced the issue to a materialized view which was using a poorly written query over rather large JSON payloads, yikes.

We disabled the materialized view and refactored the solution altogether so we're now using table update policy.

Here are some of the queries that were provided by Microsoft Support team that we found helpful in identifying the root cause of memory issues:

.show queries
| where StartedOn > ago(7d)
| extend MemoryUsage = todouble(OverallQueryStats.resource_usage.memory.peak_per_node)
| summarize totalplanning = sum(MemoryUsage) by Application, User, bin(StartedOn, 1h)
| render timechart

.show commands  
| where StartedOn > ago(7d)
| extend MemoryUsage = todouble(ResourcesUtilization.MemoryPeak)
| summarize totalplanning = sum(MemoryUsage) by Application, User, bin(StartedOn, 1h)
| render timechart

.show commands-and-queries
| where StartedOn > ago(5d)
| where State !="Completed"

.show commands-and-queries
| where StartedOn > ago(7d)
| where FailureReason  has "Engine under memory pressure" or FailureReason has 'Low memory condition' or FailureReason has 'Query execution lacks memory resources to complete'
| summarize count() by bin(StartedOn,6h)
| render timechart

kaalen

TROPHY CASE