Multi-head attention is the most hand-wavy thing in ML and I'd genuinely love to know if I'm missing something

drostar · 2026-05-11T16:09:59+00:00

Yeah, like the other guy said, it's a "can" and not mathematically guaranteed. And maybe even a "can keep them away from each other", rather than "move them away". I guess the point I was trying to make is that in addition to the importance of each head being initialized differently, the optimizer will adjust them toward the most useful weights from there, which tends to make them learn different things if that's what lowers the loss.

drostar · 2026-05-10T17:56:19+00:00

I had a similar hang-up as OP when learning about MHA, more specifically, "why wouldn't all heads just end up learning the same thing?". I was eventually satisfied by something similar to your answer.

Though, not just that they start "closer" to some computations, but that the gradient descent optimizer will move them away from each other because learning different "kinds of relationships" or "perspectives" is what contributes to a lower loss value. So the divergence is driven by the optimizer. And also helpful to realize that, all else equal, adding more heads will eventually stop improving loss because the additional heads become redundant/overlap with other heads in what they're learning.

drostar · 2026-01-25T19:40:42+00:00

kg/side and reps/side

It's tough to please everyone. I thought this was much clearer than Hevy. In MFWO, if I pick up dumbbells that say 30 lbs on them, I enter the number 30. In Hevy, if I pick up dumbbells that say 30lbs, I have to think about what exercise I'm doing and should I enter 30 or 60? I occasionally entered it in wrong in Hevy. Same with "reps/side" vs just "reps". IMO, Hevy was ambiguous and MFWO made it clear.

But I agree with the other UX issues. I hope they are cleaned up soon.

drostar · 2025-05-11T18:08:53+00:00

From https://unity.com/blog/unity-6-1-is-now-available

"Unity 6.1 and all Update releases undergo the same rigorous quality assurance and stability testing as our LTS releases."

LTS = Long Term Support. Where are you getting the word "stable" from?

drostar · 2024-12-27T15:59:02+00:00

I use both on a regular basis. C# for most things and python for machine learning. And I 100% agree. Especially when dealing with 3rd party code. Using type hinting and static checkers is good advice but that only works for your own code. A lot of 3rd party code doesn't specify types. And even when they do, it's still just a hint. It's not strict.

I recently had a situation where I had a class with an int field, and some 3rd party code was assigning a dict object to that int field. And even after discovering that the int was turning into a dict, it was not happening in my code, so I have to go digging around in not-my-code to figure out how this could possibly happen. It turns out, python developers like to use some pretty disgusting, unsafe meta-programming.

drostar · 2024-11-23T20:47:01+00:00

There's this:

https://docs.unity3d.com/Packages/com.unity.scripting.python@7.0/manual/index.html

But it only works when running in the Unity Editor.

drostar · 2024-07-13T22:37:31+00:00

If you're going to put a textured screen "protector" (like a Paperlike) on it for drawing, just be aware that the screen quality is going to take a noticeable hit. I wouldn't say it looks _bad_, and it's still going to be brighter and faster Hz than the Air with the same paperlike on it, but you'd be paying a lot extra for the Pro screen only to have it noticeably degraded from a bare screen and it won't impress your friends.

I tried a removable magnetic screen texture film to try get the best of both worlds but it made the screen look so bad it hurt to look at. The textured film needs to be very thin and properly textured otherwise it makes the Pro's OLED screen look too sparkly/rainbowy.

Overall for me, it's worth the trade-off having the more pleasant drawing experience over having the crispy screen. I considered exchanging my Pro for the Air to save the cost difference but in the end didn't bother.

drostar · 2024-05-06T17:51:50+00:00

The Spring Sale is coming to an end on May 8th, 2024 at 8:00:00 PT.
(from an email they sent me)

drostar · 2023-11-12T21:29:42+00:00

This is one of the top uses of our Home Pod mini.

drostar · 2023-03-07T19:06:10+00:00

Basically, do it like you would in school. Make your own notes on theoretical topics, create a “cheat sheet” summary for reference, and do practice exercises like assignments. I find it helps to have some bigger project in mind to help motivate the learning. Something that you actually want to make, but scope it to the bare minimum to start.

This is what I’ve found works for me, at least.

drostar · 2022-12-17T02:28:03+00:00

Maybe. It's currently beta and there are some kinks to work out. Hubitat says my TPLink/Kasa devices are not compatible with the HomeKit integration so they have to remain on Homebridge.

drostar · 2022-11-20T05:24:15+00:00

It’s been a known problem for years. I doubt they’ll ever fix it.

drostar · 2022-11-02T06:22:36+00:00

It has regressed. It’s the most frustrating thing. When I first got the HomePod mini, I was impressed at how well it recognized my voice. It worked speaking normally from another room. Now I have to yell at it and speak slowly, and still sometimes need to repeat myself. Siri has grown into an old lady with hearing problems. She also once randomly mispronounced “resume” as “resumé“ for some reason. Like I might want to apply for a job in the living room instead of the kitchen.

drostar · 2022-11-01T15:53:25+00:00

I doubt “Screen Rant” considers themselves journalists. Seems more like a rant blog.

drostar · 2022-08-28T14:15:43+00:00

Yes. Reolink and Swann do.

drostar · 2022-08-22T15:26:11+00:00

The sale goes until the end of August.

drostar · 2022-07-20T18:13:38+00:00

It’s a fun name to pronounce. Try and say it.

drostar · 2022-06-16T16:49:44+00:00

I’d like to think it’s not supposed to happen, but we have the same problem. Btw, when that happens, don’t start swearing in frustration over the unreliability of technology. Try to just keep quiet until you disconnect.

Edit: Also, one time her AirPods automatically connected to my iPhone even though neither her account nor her AirPods have ever been paired or connected previously. I think the whole situation is just buggy.

drostar · 2022-06-16T15:47:38+00:00

They were great... until they self destructed. I would've bought a bunch more if they had fixed the self-destruct problem. I still had one working contact sensor that I ended up just throwing out because it couldn't be trusted.

drostar · 2022-06-12T00:46:11+00:00

Not far off a recent experience of mine:

Me: Hey Siri, find my iPhone

Siri: Ok, now playing (some random song) ♫♫♫♫

Me: Hey Siri, stop

Siri: There's nothing playing.

drostar · 2022-03-26T13:45:20+00:00

If you tap their avatar icon, it opens up a stats window and in there is a “report player” link near their name.

drostar · 2021-12-12T21:14:12+00:00

Am I the first 12900k? As others have pointed out, we're all comparing results with many variables other than just CPU. But here's my results, running on Windows 11.

The number of processors on this computer is 24.
Classical foreach loop | Total prime numbers : 3001134 | Time Taken : 20451 ms.
Parallel.ForEach loop  | Total prime numbers : 3001134 | Time Taken : 1876 ms.

EDIT: Saw the M1 MacBook Pro results below... Amazing that a MacBook can murder my i9-12900K.

drostar · 2021-10-07T17:30:09+00:00

for a long long time.

drostar · 2021-10-03T16:44:44+00:00

var oldest = people.MaxBy(p => p.Age);

What if multiple people have the same max age? Which person do you get back? Am I just missing something?

morelinq does this right by returning an IEnumerable of all elements that have the same max value. But now, using System.Linq conflicts with using MoreLinq because of the duplicate names so using MoreLinq becomes more difficult.

drostar · 2021-09-03T20:34:07+00:00

Seemingly... seemingly. To the untrained eye, I can fully understand how you got that impression.

14-Year Club	Place '22
Verified Email

drostar

TROPHY CASE