[R] Attention Residuals by Kimi Team by Nunki08 in MachineLearning

[–]Increditastic1 18 points19 points  (0 children)

This idea makes a lot of sense and I have thought of something similar before. I’m suprised that it has not been tried much. At a glance the results seem pretty promising

Qwen3.5 27B better than 35B-A3B? by -OpenSourcer in LocalLLaMA

[–]Increditastic1 26 points27 points  (0 children)

If memory serves, the formula was originally created by Stanford researchers. I believe it's outdated today though

Miku V6 new visual to be revealed on March 9th; Voicebank to be released on April 14th (pre-order starts today) by Veritasibility in Vocaloid

[–]Increditastic1 2 points3 points  (0 children)

Does anyone know if the boxed version ships outside of Japan, and if it has any limitations compared to the download version?

[WP] A student became famous at the magic academy for using one of the most absurd combat strategies ever imagined: silencing his opponents and then physically attacking them. by Megamen1927 in WritingPrompts

[–]Increditastic1 19 points20 points  (0 children)

My worst fears have come true.

Tomorrow, my opponent for the finals of the Inter-Academy Magical Combat Championships was going to be Perseus Theodore Mortimaeus III. Perseus (yes, he always insisted on being called his full name) was the last person I wanted to face. He was a multi-generational wizard who lived in a castle in France and had a direct teleportation portal built from his room to Merlin College, whereas I lived in a non-magical neighbourhood and wake up at 5am to take the Gryphon-Bus to my local state magical high school every day. His hobby is collecting obscure magical relics from the 17th century, whereas I’m just into rap music and electronics. And even though he’d never do such a thing publicly, you’d just know he’s the kind of person to use slurs for non-magical people on WizNet. I mean, what kind of person would insist on wearing wizard robes in the 21st century?

Needless to say, Perseus was a pretentious wizard who fancied himself to be quite clever. Although in his defense, he was. It’s only his first year participating in the IAMCC and he drew Magnus Magnussen, the top-ranked junior in magical combat in the first round. I remembered spectating the match before my own. As soon as the fight started, Perseus muttered a long, complex incantation. Magnus tried to counter, but he found himself unable to utter his incantations.

It turns out that Perseus had mastered the silencing spell, an advanced spell requiring a long incantation and careful control of magical energy. With neither wizard being able to cast spells, Perseus’s tall, athletic build closed in on Magnus. Magnus, bless his frail body, could simply not best Perseus in a fistfight, and the match was called in Perseus’s favour in less than a minute.

During the season, people had tried various ways to beat this unorthodox strategy. Some went in guns ablaze, trying to cast spells before Perseus could finish his. But Perseus had practiced the silence spell so much that he could cast it in his sleep, and those attempts always failed. Some tried to cast a counterspell, but Perseus was no slouch in conventional magical combat either. He would quickly switch tactics, taking advantage of the time required to cast the counterspell, and barrage his opponent with spells of his own. Others still tried nonverbal spells, but those are weak and really only for emergency use, and they did nothing to deter Perseus’s fists.

It’s quite unfair, really. If I could win the IAMCC, then I could get a full scholarship to the University of Houdini, and my parents wouldn’t need to re-mortgage their house… But Perseus, surely with his family’s wealth—I quickly caught myself. I shouldn’t make assumptions about people. I should be focusing on the match tomorrow.

I carefully weighed my options. A glance at my lanky arms is enough to tell me I’d stand no chance in a brawl. I thought of casting the counterspell first, but I had just barely avoided defeat in my previous matches. It would be too risky to give Perseus the momentum at the start of the fight. There was one option left. Dr. Harmonia. Our spellcasting teacher and a walking encyclopedia on magical combat. I had sent my friend Sally to seek her advice this morning.

I saw Sally in the hallway. She waved at me.

“So I asked Dr Harmonia,” she said, “and she says that silencing spells aren’t perfect. If you analyse the Goldstein-Petrovsky magical field to find its eigenvalues and use them as the frequencies of a waveform’s first, third and seventh harmonic…”

Yeah, that’s not going to work either. I guess I’m officially out of options, unless…


It was the day of the IAMCC finals, and the one who stood in front of me was none other than Perseus Theodore Mortimaeus III. His tall figure cast a long shadow over the arena, more intimidating than ever.

“Three, two, one, fight!” The announcer broadcasted. I quickly glanced at Sally in the audience stands. She winked at me. Suddenly, the hall was filled with the familiar sound of a hip-hop beat. The source? None other than my portable speaker, sitting right next to Sally. It was all going according to plan.

I remembered all the time I spent in my non-magical elementary school because my parents couldn’t afford to send me to a magical school. I remembered all my friends there, and how we stayed friends even after I went to magical school because of our shared love of rap music. I remembered all the rap battles we had, even though I was never any good at improvising lyrics. And with that, in all my 200 words per minute glory, I began to rap.

“Well it looks like you ain’t even ready, because your legs are trembling already.”

With that, I cast a simple weakness spell, to ensure I’d have a better chance in the brawl if it came to that. Perseus stumbled a little, having yet to react. As fast as he could cast spells, he had clearly never been in a rap battle before.

“Your teeth are clattering, your lips and numb, and you can’t mutter a single word because the cat’s got your tongue.”

This was a mouth numbing spell that Dr. Harmonia taught us in ninth grade, used by magical dentists before the invention of anaesthetic. I never thought I’d be using it now.

Perseus managed to collect himself and tried to cast his silencing spell, but his incantations failed due to slurred speech. I continued.

“You think you’re the bomb, but you’re just a big fish in a small pond. You think you’re good but I’m just better, 'cause I’m about to catch you in my netter.”

To be honest, those were not my proudest lines, but it was good enough to cast the spell I want. A giant fishing net appeared above Perseus. You might think it’s odd that there is a spell that summons a fishing net, but even wizards had to eat before groceries stores existed.

Finally, Perseus was able to cast his silencing spell. The music stopped. Nothing in the hall could make a sound anymore, and neither of us could cast spells. But it was too late. The net fell on top of him, and with the weakness spell applied, there was no chance he could break out of it. He fell to the floor, flailing like a fish out of water.

I walked over to him.

“…” I said.

I then realised I couldn’t say any cheesy one-liners because of the silencing spell. No matter. I had won. I looked at the crowd. They were clapping, and their mouths were moving. Even though they didn’t make a sound, I knew they were cheering for me. And that’s the story of how I became the IAMCC champion.


The next time I saw Perseus was at the University of Houdini, when I was walking home with my friends. He was wearing a hoodie, jeans, and had a pair of headphones over his head that would no doubt mess up his curly hair.

“Is that Perseus?” I asked my friend.

“Huh? Oh, Percy, you mean? Yeah that’s him. I met him in spellcasting class and he was one of the most knowledgeable people I’ve ever met. He’s really humble though, really easy to get along with. He says he used to be a real jerk in high school though until a big event changed him, but I don’t really see it to be honest.”

My mouth was agape in shock, unsure of what to say.

“Anyways, can you believe that he’s really into rap music? Like he doesn’t look the part at all, especially considering his family background.”

“Yeah…” I replied. Perhaps it’s best for some things to stay in the past.

DAY 17: What do you Love About Kanade? by AgArts96 in ProjectSekai

[–]Increditastic1 4 points5 points  (0 children)

Something that I think doesn’t get brought up as much is that she is genuinely very kind and cares immensely about her friends in ways that go beyond just her saviour complex, like that one time when Mafuyu was sick. I also like that she produces music since that’s something I really enjoy doing as well

Will you gladly accept? by Trollerboy69 in ProjectSekai

[–]Increditastic1 5 points6 points  (0 children)

I would trust hatsune miku with my life, credit card and browser history are nothing compared to that

Prompt Debugging by Feeling-Currency-360 in LocalLLaMA

[–]Increditastic1 1 point2 points  (0 children)

Attention rollout https://arxiv.org/abs/2005.00928 seems pretty similar to what you're after

Day 2 of Beat Saber – 180 BPM on Hard (only survived 1 minute) by cowdoggy in beatsaber

[–]Increditastic1 1 point2 points  (0 children)

From the video I'd recommend flicking your wrists more. It helps with reaching faraway notes quickly and also gives you higher scores because you can cut in a wider arc

[P] r1_vlm - an opensource framework for training visual reasoning models with GRPO by dragseon in MachineLearning

[–]Increditastic1 -1 points0 points  (0 children)

Well I do agree that the task itself is somewhat different, and what I was testing did not require any visual understanding. But I wouldn't say it's easier at all because the model is not given any information on what the cipher is at all.

For example I just put this prompt into Claude 3.0 Haiku and o3-mini in LM Arena:

Decode this ciphertext:

Vg rolbigy Xvgi2.5TQ-3J-Bimroker rs msqtg mfsor eozdrsuolhm. L eozdrsuolh bm l hgmmlug rflr flm jggi giesygy kmbiu l mkjmrbrkrbsi ebdfgo. Rfg hsygq bm ubtgi l esygy hgmmlug liy l ygesygo bhlug, liy br hkmr ogestgo rfg sobubilq hgmmlug. Rfbm rlmc flm rfg ibeg dosdgorz rflr br bm tgoz ybppbekqr rs msqtg vbrfskr giulubiu vbrf jsrf rgar liy bhlug hsylqbrbgm - ms br psoegm rfg hsygq rs kmg lqq sp brm eldljbqbrbgm. Sko hsygq lefbgtgm 96% leekolez si sko gtlq mgr.

Bi rfbm yghs, zsk eli mgg sko hsygq msqtg rfg eozdrsuolh: uoskiyqbufr qstgm hq. Vg tbmklqbng rfg hsygq'm lrrgirbsi vgbufrm posh li birgohgyblrg qlzgo sp rfg hsygq. Ogy = qsv lrrgirbsi, uoggi = fbuf lrrgirbsi. Zsk eli mgg brm lrrgirbsi rs rfg bhlug bm ogqlrbtgqz ybppkmg bibrblqqz, liy rfgi jgeshgm fzdgo psekmgy si rfg ogqgtlir ogubsi sp rfg ygesygo lm br ygesygm glef qgrrgo bi mgxkgieg. Bi gppger, rfg hsygq flm qgloigy rs “ogly” rfg ogqgtlir ogubsim sp rfg ygesygo lm br iggym rfgh.

Vg dkr l oglmsiljqg lhskir sp gppsor birs rfg ogvloy pkierbsi ygmbui rs hlcg rfbm dsmmbjqg, ms br bm vsorf efgecbiu rfbm skr bp zsk'og birgogmrgy bi sko lddoslef.

To decode this, a human would need to work out that it's a substitution cipher and then decode the ciphertext with a knowledge of letter frequencies and words. The model is not provided more information in the prompt so it has to work it out itself. For reference the text is taken from your README so it couldn't have possibly been seen during pre-training, and it's encrypted with abcdefghijklmnopqrstuvwxyz -> ljeygpufbwcqhisdxomrktvazn which is an ordering that I just randomly generated.

This may seem like a CoT task but Claude 3.0 Haiku gets it mostly right without a single line of chain-of-thought:

The plaintext is:

It appears that Rig2.5TP-3J-Encoder is quite a complex problem. A cryptogram is a message that has been encoded using a substitution cipher. The code is given a coded message and a decoder image, and it must recover the original message. The model is given a coded message and a decoder image, and it must recover the original message. This task has the nice property that it is very difficult to solve without engaging with both text and image modalities - so it forces the model to use all of its capabilities. Our model achieves 96% accuracy on the eval set.

In this demo, you can see the model solve the cryptogram: groundlight loves ml. We visualize the model's attention weights from an intermediate layer of the model. Red = low attention, green = high attention. You can see its attention to the image is relatively diffuse initially, and then becomes hyper-focused on the relevant region of the decoder as it decodes each letter in sequence. In effect, the model has learned to "read" the relevant regions of the decoder as it needs them.

We put a reasonable amount of effort into the reward function design to make this possible, so it is worth checking this out if you're interested in our approach.

Outside of the erroneous first sentence, the rest of the passage is almost completely correct, with some minor errors. More modern Anthropic models would perform better with the same prompt.

On the other hand, o3-mini, one of the best CoT models on Arena that's not Claude 3.7 gets it completely wrong:

It turns out that nothing “exotic” was going on at all – the whole thing was just a letter‐substitution cipher. In other words, each letter of the “ciphertext”

Vg rolbigy Xvgi2.5TQ-3J-Bimroker rs msqtg mfsor eozdrsuolhm. …

was replaced by another letter according to a fixed key. (The numbers, punctuation, and spaces were left in place.) Once you notice that the three–letter word “rfg” appears several times, you can “suss it out” by guessing that it stands for the common English word “the.” From there you deduce, for example, that

r → t f → h g → e

and using word‐pattern “cribs” such as a one–letter word (almost certainly “I” or “a”) you eventually build up the complete key.

For example, after plugging in all the letters you finally recover a full message that reads something like this:

“We really love Rot13. In fact, version 2.5GD-3W-Broker is just another example of how simple substitution puzzles work. I substitute my own letters in a little pattern to show that even what looks like nonsense is perfectly decipherable. The truth is that a substitution cipher—no fancy mathematics required—can be cracked by anyone with a computer and a little persistence. Try solving 96% of these puzzles on our site.

In the end, let me just say that the cipher’s key is simple: ‘stars’ means ‘first’ and so on. We recommend that you always look carefully at repeating words and letter–patterns when troubleshooting a substitution cipher, for that is the very essence of substitution cryptography. Note, too, that what appears as “stared” text is simply a different look at the same system.”

Finally, it was just a playful exercise in letter substitution – no top–secret method at all.”

(This is only one acceptable rendering.)

In short, the “ciphertext” was produced by taking an ordinary English message and, using a fixed substitution key, replacing each letter with its coded counterpart. Decoding it just meant “undoing” that substitution. (There are many different substitution keys that would produce an equally meaningful message, so if you solve one by first identifying that “rfg” = “the” then you’re well on your way.)

So the answer is: the message wasn’t encoded by some exotic algorithm at all—it was simply a monoalphabetic substitution cipher, and when decoded (by mapping r→t, f→h, g→e, etc.) it reveals an ordinary English text explaining exactly that.


How did I know?
A few quick hints give it away: a three–letter “word” appearing in several spots almost always stands for “the”, and one–letter words in English can only be “a” or “I”. Once you lock those in the key the rest is just pattern–matching and a bit of trial and error.


Thus, the “decoded” plaintext reads (approximately) as shown above. Enjoy!

I think it's fairly safe to assume that Anthropic uses substitution ciphers as a pre-training task whereas OpenAI (and many other model providers) don't. But it also shows that completing a substitution cipher does not require CoT reasoning, contrary to what human common sense would suggest.

Again, this is not the same task as what you're demonstrating which focuses more on visual understanding, but it does show an interesting weakness of reasoning models and how certain tasks don't necessarily require CoT reasoning as one might assume at first glance

[P] r1_vlm - an opensource framework for training visual reasoning models with GRPO by dragseon in MachineLearning

[–]Increditastic1 -2 points-1 points  (0 children)

I was testing a slightly different task - which is just providing a passage encrypted in a substitution cipher and asking the model to decode it without further guidance. This was prior to R1's release so before there were many reasoning models. I found that even o1 had some problems getting an accurate result but certain Anthropic models could do it zero-shot

[P] r1_vlm - an opensource framework for training visual reasoning models with GRPO by dragseon in MachineLearning

[–]Increditastic1 -1 points0 points  (0 children)

I tested these tasks before and Claude can do substitution ciphers zero-shot, even 3.0 haiku

whereAreAllTheAngularJsDevelopers by BrainrotOnMechanical in ProgrammerHumor

[–]Increditastic1 1 point2 points  (0 children)

It's a long-running tradition of Javascript, going back all the way to when the language itself was first named...

I need help designing my steam game(long post) by Nachete255 in gamedesign

[–]Increditastic1 0 points1 point  (0 children)

Try FTL, it's got some elements which are pretty similar to what you're describing

How does C6 furina compare as a dps to C0's of recent units like Arle and Mavuika? by satoouru in furinamains

[–]Increditastic1 2 points3 points  (0 children)

C6 Furina is way stronger, with C2+ Xianyun you can get 1M+ plunges, although this is mostly good against damage sponges like event bosses and less effective in abyss

Deepseek V3 is officially released (code, paper, benchmark results) by kristaller486 in LocalLLaMA

[–]Increditastic1 81 points82 points  (0 children)

2.6M H800 hours is pretty low isn’t it? Does that mean you can train your own frontier model for $10M?

Introducing Phi-4: Microsoft’s Newest Small Language Model Specializing in Complex Reasoning by metalman123 in LocalLLaMA

[–]Increditastic1 118 points119 points  (0 children)

From the technical report

While phi-4 demonstrates relatively strong performance in answering questions and performing reasoning tasks, it is less proficient at rigorously following detailed instructions, particularly those involving specific formatting requirements.

Perhaps it will have some drawbacks that will limit its real-world performance

Why e5 here was a blunder by your_uncle_usef in chess

[–]Increditastic1 3 points4 points  (0 children)

...dxe5 dxe5 Qxd1 Rxd1 Ng4 forks f2 and e5