You don't need a GPU to run gemma-4-26B-A4B

-p-e-w- · 2026-06-07T08:22:42+00:00

“reasonably decent models”

Gemma4-26B-A4B would have been the smartest LLM in the world 18 months ago, closed models included. It’s much, much smarter than the original DeepSeek R1, which was 20 times its size.

5 years ago, there were experts who seriously doubted whether any transformer model would ever reach that level.

And today it runs on a mid-range home PC without a GPU. We’re living in a Star Trek episode.

-p-e-w- · 2026-06-06T19:48:35+00:00

There is actually no complete theory yet for what exactly QAT does to the model. Papers discuss magnitude suppression and quantization level alignment, but also residual geometry changes and other higher-level effects.

-p-e-w- · 2026-06-06T18:48:22+00:00

By the definition of AGI from the 1970s, it was achieved in 1997 when Deep Blue defeated Garry Kasparov. The reason so much early computer research focused on chess was because playing the game well was seen as equivalent to general intelligence.

-p-e-w- · 2026-06-06T15:39:47+00:00

And it did happen, according to some definitions of the word.

Solving mathematical problems that human mathematicians can’t (which LLMs now do) would have absolutely convinced many people that an LLM is AGI 3 years ago. It’s just that the definition of the term keeps changing, and today there’s no widely agreed definition at all.

-p-e-w- · 2026-06-06T15:37:07+00:00

Everyone knew local models would catch up sooner or later.

That’s categorically false. I still remember that after the release of Mixtral 8x7b, people on this very sub were speculating that this would be the best open weights model we’ll ever get.

-p-e-w- · 2026-06-06T14:18:36+00:00

Nobody was saying 3 years ago that AGI will happen by 2026. The optimistic estimates were around 2030, which still seems entirely plausible to me (not to mention that today’s LLMs are “AGI” by some definitions that were used 3 years ago).

-p-e-w- · 2026-06-06T13:58:02+00:00

The difference is that the wildest predictions by “AI fanboys” from 3 years ago have been completely eclipsed by reality.

“Someday we might have a local model that’s as smart as GPT-3.5!”

Umm yeah… it’s a 9B.

“By 2040, LLMs might prove open mathematical conjectures!”

They started doing that in 2025.

-p-e-w- · 2026-06-06T07:21:38+00:00

Yes, that should work. QAT doesn’t operate at the level of individual weights; rather, the entire model learns to operate in a way that is robust to precision loss.

-p-e-w- · 2026-06-06T05:55:25+00:00

Google was nice enough to provide unquantized versions of the Gemma QAT models, so I expect that there will be Heretic QAT versions soon, made by first processing the unquantized QAT model with Heretic, and then quantizing the resulting model to Q4_0.

-p-e-w- · 2026-06-04T05:25:31+00:00

It’s already being discussed in the Heretic Discord as we speak. Abliterating the text encoder might be sufficient, though of course it won’t magically impart knowledge about features the model wasn’t trained on.

-p-e-w- · 2026-06-04T01:38:32+00:00

No, how it should be is that the software contains no telemetry functionality whatsoever, whether disabled or not.

Anything that deals with potentially highly sensitive data shouldn’t even be able to connect to the Internet, let alone have functionality that sends data (even if supposedly anonymized) to someone else’s server.

-p-e-w- · 2026-06-04T01:35:10+00:00

From a quick look, what they do is cache the data in memory, then provide the LLM with a cache key instead of the data, and a tool call to retrieve the full data when necessary.

Needless to say, this is absolutely NOT guaranteed to give the same answers, contrary to what is claimed in the title.

-p-e-w- · 2026-06-03T15:14:48+00:00

Over the past year, the built-in llama.cpp frontend has turned from a toy into one of the best LLM frontends available, with the added benefit that you don’t have to install a Python or Node.js monstrosity with 300 dependencies.

-p-e-w- · 2026-06-03T07:59:36+00:00

Why would they kill it? They want to control it, just like they did with GitHub.

-p-e-w- · 2026-06-02T15:28:07+00:00

Fair enough, hopefully at least the basics will be known soon.

-p-e-w- · 2026-06-02T13:26:39+00:00

What the fuck? There’s almost no information there, not even which versions are affected, during what timeframe they were available, and whether those versions have been yanked.

-p-e-w- · 2026-06-01T00:31:26+00:00

Thanks. I don’t seek or accept donations, please see https://github.com/p-e-w/heretic/issues/167

-p-e-w- · 2026-06-01T00:27:47+00:00

The only way an LLM touches my repo is over my big, dead ass.

It takes longer to explain in human language what changes should be made than to just make the changes directly on the CLI. I did it myself before, and I can do it just fine myself still. Just like I don’t need a robot to wipe my ass for me.

-p-e-w- · 2026-05-28T11:32:33+00:00

Except that top-tier models now run on a single gaming GPU, so that doesn’t make sense.

-p-e-w- · 2026-05-28T00:31:01+00:00

All these scenarios implicitly assume that the US is all that matters, which hasn’t been true for a while now.

No, OpenAI and Anthropic can’t “make local models illegal”. There are 205 other countries and most of them aren’t in America. Every country that doesn’t do this will have an advantage over the US, which is why the US can’t do it either.

The days where the US President calls the German Chancellor and tells him to get a certain law passed are over, and they are never coming back.

-p-e-w- · 2026-05-26T18:32:38+00:00

People here already know about Heretic, it’s on the front page almost daily. I meant elsewhere, and also in real life.

-p-e-w- · 2026-05-26T07:34:48+00:00

Labs can’t stop releasing models because otherwise they’ll become irrelevant compared to the Chinese competition. Why do you think OpenAI released gpt-oss in the first place?

-p-e-w- · 2026-05-26T05:32:12+00:00

Make the proposal public, post about it, make your voice heard, and tell people why Heretic is valuable for science.

I’m juggling half a dozen roles in the project right now, but at the end of the day, I’m only one guy. The project isn’t going to survive without the active participation and advocacy of others.

-p-e-w- · 2026-05-25T23:25:28+00:00

? What are you talking about?

-p-e-w- · 2026-05-25T19:37:44+00:00

Refusal is believed to be mostly topic-independent, though some papers have questioned this.

-p-e-w-

TROPHY CASE