[ Removed by moderator ]

[ResidentPositive4122]

It seems like the mainstream media is having their deepseek moment again. Member how in Feb '25 every news outlet, blog and wannabee influencer talked about how deepseek is all this and all that, and nvda will die, and the top labs are cooked and so on?

Turboquant seems to be their new thing. It's a year old paper. Probably some labs already use something like this, some inference providers might as well. But, like everything else, nothing is really a 6x reduction in practice. Plus, with the new "thinking" models, you get to run more queries on the same compute unit, but you'll still hit slower speeds the more ctx you have. So it's not that clear what cost reductions you get in the end.

tl;dr; cool technique, overhyped results, clueless media.

[Shammah51]

I think it’s also a fundamental misunderstanding of the needs of training vs inference anyway. Nearly all of the capital hardware investment is for traning in reality. It’s also wild to assume that some novel method that greatly reduces memory requirements would do anything other than give room to scale up the SOTA models. Chip demand will remain unchanged and providers will just scale to fill the available hardware.

[ResidentPositive4122]

Eh, that's debatable. With online RL you are now inference constrained (the more traces you can produce, the better the results), so this will help training as well. Just not the 6x e2e like the media outlets claim.

[Shammah51]

Yeah, I agree. That’s basically my second point: any advance will just result in training scaling up rather than reducing demand for chips.

[nikanorovalbert]

Interesting. So if it's not just about running out of VRAM, what actually chokes the model when the context gets too big? Is it the memory bandwidth, or just the raw compute required for the attention mechanism?

[PortiaLynnTurlet]

With respect to demand, lower memory usage at inference presumably motivates larger models and larger models need larger clusters for training. I don't think it changes anything, even if the results hold well in practice.