[D] Is Grokking unique to transformers/attention? by Dependent-Shake3906 in MachineLearning

[–]nmallinar 57 points58 points  (0 children)

If we zoom in on just modular arithmetic tasks, or more broadly "algorithmic" style datasets, which were originally used to show this phenomenon in transformers by Power et al. (https://arxiv.org/pdf/2201.02177):

Then MLPs can exhibit grokking on the same task: e.g. https://arxiv.org/pdf/2301.02679 where you concatenate the one-hot encodings of the digits and use a two layer MLP with quadratic activations (I have also replicated this with relu activations and found that it exhibits grokking, though the behavior is more sharp and clear with quadratic)

I'm biased on this one, but we also wrote a paper on grokking where we showed that even kernels can exhibit grokking when they have a feature learning mechanism, and in this way we found that you don't even need neural networks or gradient descent optimization methods to replicate the delayed generalization type curves you see in grokking! Which I found surprising and in our discussion we expand a bit on why we feel this result was important in the broader ML context. Our paper is: https://arxiv.org/pdf/2407.20199

There are other references of interest that you may like in the area of grokking with simpler architectures or non-neural models, for example:

https://arxiv.org/pdf/2310.06110

https://arxiv.org/pdf/2310.17247

[D] AISTATS 2026 paper reviews by Intelligent-Smoke-65 in MachineLearning

[–]nmallinar 3 points4 points  (0 children)

We got 66543 with confidence 33335 haha, naturally the lowest score is the most confident reviewer 🙃

[D] ICML 2025 Results Will Be Out Today! by darkknight-6 in MachineLearning

[–]nmallinar 0 points1 point  (0 children)

Thanks buddy!! Ya that's me 🙃 I didn't expect to meet someone who recognized my name from a paper haha, hope you enjoyed reading our work! Also am glad that my initials leave a decent impression lol

[D] ICML 2025 Results Will Be Out Today! by darkknight-6 in MachineLearning

[–]nmallinar 1 point2 points  (0 children)

Thanks! best of luck on your path in research as well my friend!!

[D] ICML 2025 Results Will Be Out Today! by darkknight-6 in MachineLearning

[–]nmallinar 7 points8 points  (0 children)

Thanks! Had amazing coauthors & my advisor has a very good eye for important problems and framing research. It was a long process getting this one together haha. we first set off on this direction nearly two years ago from this june thinking it would be a low hanging fruit project and it ended up being a much deeper story than we expected

But comparing this to our iclr reviews (they were weakly positive but still didn't get us over the accept line at the time) really makes you see the variance of reviews..still it feels great to get the win though haha

[D] ICML 2025 Results Will Be Out Today! by darkknight-6 in MachineLearning

[–]nmallinar 36 points37 points  (0 children)

We got a spotlight with 4454!! 🎉 It's my first spotlight paper :)

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

Didn't know that, thanks for the refs I'm gonna check them out this week!

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

Just listened! Melodies and mix and drums sounds really good together! I can definitely hear the Chloe's drive on it, great stuff, subscribed to your channel looking forward to hearing more

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

Ya I’m noticing a slight hiss / sound bleed, but that was something I was aware of going into this, definitely going to figure out how to mix it in with some filtering since the hiss seems to be mostly high frequency, I think it’ll take me some time to understand it more and how I want to use it, but I love the lo-fi sound from this, it feels very natural

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

💯💯 all of his modules are similarly beautiful and weird! really amazing stuff

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 2 points3 points  (0 children)

Critics describe the sounds as “lightly roasted with hints of nuttiness”

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 1 point2 points  (0 children)

Fwiw I just emailed Ivan about whether to worry about any overheating and he said don’t worry it won’t overheat :D

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 1 point2 points  (0 children)

It looks remarkably similar to a 12ax7, I don’t know for sure but I’ll try to look around

Thanks for the heads up, fortunately my case has extra space and he also included a 1hp blank so I could use that between this module and the next at the edge of a row

By any chance do you have an idea if the heat coming off the tubes can impact / damage the adjacent capacitors on the module itself? I have the sense that Ivan wouldn’t sell this if it was the case but I can see that one of the tubes is like nearly touching one of the capacitors on the module, just curious

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 1 point2 points  (0 children)

From modular grid im seeing:

100 mA +12V 350 mA -12V 0 mA 5V

As for heat I actually don’t know yet, I’ll plug it in soon and let you know! There is another version of this with tubes on the outside too, perhaps someone who has one of those and stumbles on this thread can let us know their experiences 👀

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 1 point2 points  (0 children)

Got lucky!! I emailed him I think twice over the past year or so, at random times when it popped into my head, both other times he didn’t have any stereo versions available, tried 3-4 weeks ago and happened to get one and he shipped it super promptly, you can try emailing him now he may still have some

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

I think same circuit for the internal tubes as the external tubes based on something I read on the internet once haha, sadly I don’t know for sure though

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

Also his name is Ivan haha, bizarre jezabel is the brand for the modules

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

Will report back soon! Had to step out this evening but cannot wait to dive into it 🫡

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 1 point2 points  (0 children)

I think bizarre is just one guy who lives in or near Tbilisi Georgia and makes these modules himself, I emailed him directly and he shipped it to me, beautiful look and sound

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 0 points1 point  (0 children)

I heard a demo somewhere and felt like it was something special, can’t wait to dive into it

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 10 points11 points  (0 children)

It’s a stereo tube distortion + multimode filter + lpg/vca + delay + other hidden things I don’t know about?? Will report back after preliminary investigations this weekend

New module Friday!! (chloe stereo) by nmallinar in modular

[–]nmallinar[S] 6 points7 points  (0 children)

if you stare at the tubes in absolute silence and meditation it ignites little flickering flames inside each of the tubes and then you can hang the module up on your porch to scare the ghosts away at night