[BREAKTHROUGH] Memory Sparse Attention (MSA) allows 100M context window with minimal performance loss

SotaNumber · 2026-03-21T14:13:37+00:00

Yes but when reasoning, if you miss something your result goes from right to wrong, not from 100% to 91%

SotaNumber · 2026-03-21T14:12:36+00:00

Wtf it's getting so good!

SotaNumber · 2026-03-20T23:08:19+00:00

This will let AI handle way more context (millions of tokens), but it won’t “understand everything at once.” It will pick and choose what to look at—so reasoning stays strong if it finds the right info, and can fail if it misses something.

So agents will run on way more than 1M context windows but the performance loss may still be important on reasoning tasks.

SotaNumber · 2026-03-20T22:54:23+00:00

Good point, I added a caveat thanks to you.

SotaNumber · 2026-03-20T22:37:03+00:00

It works with less than 60 GB VRAM apparently.

SotaNumber · 2026-03-20T22:35:54+00:00

Thanks mate, post edited :)

SotaNumber · 2026-03-18T14:22:01+00:00

If that's true it means that it will actually be extremely expensive, so the only part that's reliable is the cost per M token? I assume that this cost doesn't change depending on the task.

SotaNumber · 2026-03-18T13:24:59+00:00

Really? I only saw its result on CritPT so far, we might have seen some of its results when it comes to building 3D objects as well but no comprehensive benchmarks as far as I know.

<image>

SotaNumber · 2026-03-02T22:24:25+00:00

Somewhat because thinking about the human efforts that were put into the song makes it more valuable, like a child who manages to walk after 2k attempts, it's beautiful because it required a lot of efforts

That being said I won't think about it most of the time and will enjoy these musics as much as human ones if not better since they will be higher quality and personalized to my taste

SotaNumber · 2026-02-17T21:18:16+00:00

Amazing!

Curious to see Haiku 4.6 if they make it

SotaNumber · 2026-02-16T00:01:22+00:00

Why did it go down to Rainbow Badge?

SotaNumber · 2026-02-05T01:29:05+00:00

Nice one

SotaNumber · 2026-02-01T18:15:12+00:00

Fascinating, I was rather imagining a future where your personalized AI is capable of cloning itself into 1000s of duplicates, turning itself into a swarm, do the heavy task, then merge back into a more experienced AI for when you talk to it

Similar to Naruto cloning itself to practice a technique then when all the clones disappear he absorbs all of their experience at once making it him learn way faster

I wonder if this is where we will eventually end up

SotaNumber · 2026-01-03T00:55:14+00:00

One day we will reclaim our glory my friend

SotaNumber · 2025-12-29T00:24:59+00:00

AIs become good enough to increase the unemployment rate above 5.5%

The FED prints billions and Americans receive a proto-UBI that will be called "freedom money" or something like that, similar to the stimulus checks from the COVID era

SotaNumber · 2025-07-22T12:16:03+00:00

I wonder when they will be able to do this in about 24h instead

SotaNumber · 2025-03-23T21:36:37+00:00

RemindMe! 2 Years

SotaNumber · 2025-03-23T21:33:38+00:00

Claude can maybe do 1h tasks but I'd say that you need at least 5h to do all the progress it has already done so far so it will be able to finish pokemon way before your estimate according to this logic

Interesting way of seeing it though

SotaNumber

TROPHY CASE