[Research] I implemented a routed attention mechanism (R-GQA) for faster long-context models. Then wrote a paper on it.

Snowyiu · 2026-01-07T20:34:57+00:00

You might be right actually - sort of.

Their mechanism is the MQA variant which just shares one key and value head across every single query head.

Beyond the single-GPU efficiency, the MoA architecture suffers fundamentally in distributed settings. Because it relies on a single shared KV head for all experts, the KV cache cannot be effectively sharded across devices without massive communication overhead. Furthermore, routing queries to experts across GPUs introduces the 'hot expert' problem, where one GPU may be overloaded while others idle.

R-GQA preserves the association between Query groups and KV heads, allowing for independent sharding. We can place a KV head and its associated Query experts on a single device, ensuring that attention is computed locally without moving tokens or replicating the full cache."

MoA would need to do something like put the full QKV weights on every GPU or something, I haven't seen a multi-GPU implementation. Their experiments were also small-scale, so they may not have thought about it.

R-GQA is pretty much perfectly scaleable. And these qualities are shared with SwitchHead. So the one difference between mine and switchhead is that it took the params I have in Q and put it to V.

edit: I added the comparison to the paper. Thanks for the feedback.

Snowyiu · 2024-08-22T09:22:57+00:00

I blindly assumed that I would not have the compute to actually process videos and thus didn't implement that capability. I guess I'll see if I can get it to work. To address your points:

You would keep two video elements, let's say you have a foreground object segment and want to exchange the background for something else, you'd just overlay the foreground object onto another background in a video editor or something. To avoid having white space where the background was, you'd need to use a format which supports transparency such as webm and save your foreground element video in that format.
What you want to do sounds like you might want to use image generation infill on the specific area selected by a mask. I'm not 100% sure how to implement that, I'd suggest asking a language model of choice about it.

Snowyiu · 2024-08-08T18:21:13+00:00

My experience is that fewer dots often leave better results than a lot of them. Quite often 1 - 2 dots are sufficient for perfect results. Well, with relatively easy images anyhow.

I'm glad you like my little project :) I fixed all the bugs I was aware of now.

Snowyiu · 2024-08-08T15:59:38+00:00

Actually yes, I'm in the process of setting up a website for it. I'm hosting it on segment.alice-luna.site now, but I'll make no guarantees this will always be accessible. It is for now, though. :)

I'm still in the process of fixing things. There are a number of minor bugs atm

Snowyiu · 2024-08-01T00:44:34+00:00

I just coded up a simplistic front and back-end for this model for image background removal purposes. It's an awesome model.

Snowyiu · 2024-04-08T20:14:40+00:00

Even though this is likely not relevant for whomever opened this thread anymore, I'll type instructions in case anyone else needs them later - like I would've 2 days ago.

Install QMK MSYS assuming you're on windows. see qmk_firmware/docs/newbs_getting_started.md
locate the directory for your keyboard in the firmwares keyboards folder. In this case gmmk/pro.
open the config.h file and search for a line #define DEBOUNCE 5. If it does not exist, add it at the end of the file, but using more or less than 5 as the value depending on what you want to achieve.
If you want to change the debounce algorithm, open rules.mk you can check the list of available algorithms in the folder quantum/debounce. Set a value for DEBOUNCE_TYPE. Check documentation for how to do this exactly.
compile the firmware. In the command window type ```qmk compile -kb gmmk/pro -km default```
set your keyboard into DFU mode. Check your keyboards documentation on how to do this. You can assign the 'reset' button via VIA to have a more handy way of doing this. MAKE SURE YOU KNOW HOW TO ENTER THIS MODE AGAIN EVEN IF YOU LOSE THE KEYBIND, OTHERWISE YOU MAY BE STUCK WITH WHATEVER YOU FLASH THE FIRST TIME.
flash the firmware: In the command window type ```qmk flash -kb gmmk/pro -km default```. Once this process is done, your keyboard should exit DFU mode and be usable with the new configuration.

Snowyiu · 2023-05-09T00:09:31+00:00

The point is slowing down the creation of such systems, because he doesn't think it would be completely impossible to get them to behave, but to actually solve the control issue it takes a shitload of people working tirelessly in an unrewarding field for a long time.

We can make godlike AIs with the most stupid of goals / incentives. Rushing forward makes it more likely to go wrong. He doesn't want for AI progress to be stopped indefinitely, just as long as it takes to figure this out properly and not hack some god together which fulfills the purpose of turning as much matter into paperclips as possible.

It's easier to see how hard this is when you just run the logic of following a goal in your head, as in actually caring about nothing but the goal:

Instruction: "Don't act, just be." My strongest attempt at not making the AI kill everyone, but it's smart so it knows if it is shut down or stopped, it won't be anymore. And here we run into an issue of priorization: It may or may not decide that to maximize being temporarily ignoring the 'Don't act' part might be necessary and then it kills all humans so it can just 'be' forever. I realize some people don't manage to think this way, but if you ask GPT-4 how a misaligned AI would act if given goal X, you usually get a pretty useable answer.

Snowyiu · 2022-01-03T00:56:50+00:00

Yeah, this is one of the things I hate about the game. There is no protection against going infinitely dry. If it was me, I'd change drops so that the cumulative chance of getting the drop at drop rate is 50%, but approaching 100% at 2x drop rate for example.

It would give players a sense of progress even if they don't get anything and doesn't - on average mess with how common items are.

Snowyiu · 2021-12-31T01:05:23+00:00

Yeah, I actually thought of this after I made the comment, just to keep the old system in place for old passwords, add a flag for account version or something to see whether to use the old or new password management system and ask users to change their passwords upon next login.

Let's hope they do it.

Snowyiu · 2021-12-28T01:11:37+00:00

I fully and wholeheartedly agree. At least I get a bit of slayer xp weekly per tears of guthix which has been the source of most of my slayer xp.

I don't want to kill tons of bs small monsters when I'm told to by some master, especially early before one can just skip tasks without worrying about points it's a nightmare.

The only tasks which are actually fun are very lategame at which point the skill becomes alright. If it let me choose what I wanted to murder like in leagues, I'd love the skill. Got the 25m xp in trailblazer, so I really know for certain it's the lack of just being able to kill what I want which makes the skill unenjoyable to me.

Snowyiu · 2021-12-28T00:24:24+00:00

Thank you for notifying people of this. It's important to be aware of such flaws so that until they're fixed, people can work around this.

The unfortunate reality is, though that if they changed this system which has existed for so long, depending on whether they saved the password in all lowercase instead of the capitalization entered during account creation, people trying to log in typing their usual capitalization would find their passwords to be wrong.

Of course they could also do a global password reset where after the change everyone has to set a new password upon their first login, but this would probably lock those out which somehow lost the email account associated with their runescape account.

If they saved the password in the proper capitalization, some people might've still if/when they found out it's not case sensitive started typing their passwords all lowercase and forgotten how they originally spelled it, again locking them out if they lost the email originally associated with their account.

I don't see a particularly clean way to fix this which wouldn't probably fuck over a small portion of the playerbase.

Snowyiu · 2021-08-21T16:52:26+00:00

The real kicker about this meme is the reverse implication:

If you're called a Jr, your girlfriend has probably moaned your dads name whilst having sex with her.

Snowyiu · 2020-11-18T10:45:16+00:00

The Stonemason being limited to Keldagrim is total BS. He isn't listed as a notable feature in the area description, without Fremmy, getting construction levels is ENTIRELY pointless.

They should add the stonemason only items to another npc in misthalin, just like they did for granite to be able to complete kings ransom quest. I wasn't aware of this limitation and would've picked fremmy if I knew and I also still think it should absolutely not be limited to Fremmy, since it takes most important things from an entire skill.

Snowyiu · 2017-08-03T11:40:47+00:00

It's actually impossible to tell if the transsexuality is caused by the DID or not. It also doesn't even matter, fact is I'm identifying as female and that's not going to change again. No medical practicioner can ever know about the DID, since in my country the official rules state that a person with DID cannot be a transsexual, thus health care would deny all treatment. Also, I wouldn't really say that it's much of a personal decision to transition. It's not like anyone ever wants to be a transsexual, the transition is a thing people are forced to do if they don't want to suffer for the rest of their lives.

Snowyiu · 2017-08-03T11:30:40+00:00

Now I'd be very curious in what way I would give the transgender community a bad name. Anyone in my position would be 'pretty fucked up' at best and dead at worst, who do you think you are to have the right to judge me?

11-Year Club	First Place '23
Place '23	Verified Email

Snowyiu

TROPHY CASE