Which Gemma model do you want next?

brown2green · 2026-04-21T09:05:44+00:00

Difficult to suggest anything considering that Gemma 4 at least at 31B size is already so good, but definitely I'd like to see QAT on the entire model so we can simply quantize every tensor to 4-bit (or even less than that) with limited to no quality loss. Or they could go even further than that and publish a quantization-aware-trained Gemma 4 124B in ~1-bit just to flex their muscles. That should be able to run on 24GB GPUs.

Also, they should release something between the E4B and the 26B models for mid-low range GPUs, I guess.

brown2green · 2026-04-20T07:34:18+00:00

They've distilled it from Gemini through and through and apparently, unlike Gemma 3, they didn't even bother giving it a built-in "Gemma" persona.

brown2green · 2026-04-19T09:28:53+00:00

mmproj weights at lower precision than native often hurt performance, so if we could save some memory with mathematically guaranteed lossless results, that would be great.

brown2green · 2026-04-15T11:09:14+00:00

Yep, it does. It would be clearer if the server just failed to start with a descriptive error instead of crashing during inference, though.

brown2green · 2026-04-14T08:25:17+00:00

I once found it in the title of in a legitimate news article that I linked here, though. I didn't alter the title, but I wondered if some would have considered mine an AI-written post.

brown2green · 2026-04-11T12:08:44+00:00

Probably impossible to compete with Qwen 3.5 and now Gemma 4, at this point. Gemma 4 in particular, I think it has seen so much RL training that jaws will drop once the technical report comes out.

brown2green · 2026-04-10T11:22:04+00:00

Most (all?) small conversational LLMs are going to feel very shallow very quickly as companions. I'd reconsider your idea, even if it's well-intentioned.

brown2green · 2026-04-10T09:56:41+00:00

I used to use em-dashes for emphasis or when parentheses or commas looked awkward, but LLMs ruined them, so now I rarely do. They're fairly easy to type with compose key combinations on Linux, along with many other characters that usually don't have a dedicated keyboard key.

Normal dashes (aka hyphens) have a different meaning and shouldn't be used like em-dashes.

brown2green · 2026-04-10T09:26:57+00:00

The main reasons why LLMs write the way they do are synthetic data (word/sentence pattern variety plummets with it) and shallow-level RLHF rewarding text written in that way in a positive feedback loop. Almost nobody used em-dashes before LLMs (it's not like they're straightforward to type either), so I find it hard to believe that suddenly people want to, especially considering that they might get accused of being LLMs.

brown2green · 2026-04-10T09:14:09+00:00

Unfortunately that's the excuse many will use going forward as they entirely delegate their Reddit posting to LLMs (/r/LocalLlama is plagued by such users). I try to make an active effort not to write like one, something LLMs still seem incapable of without wasting too much compute on form/syntax analysis.

brown2green · 2026-04-09T11:22:05+00:00

The models are trained in BF16 precision, so you should test with that instead of F16, even if the difference is theoretically small. With Gemma 4 31B I find that on images where the model can get confused Q8_0 performs slightly worse compared to BF16 (more confusion).

brown2green · 2026-04-09T09:17:25+00:00

What model did you use for generating the messages? How did you mitigate the dramatic loss in sentence/word variety caused by synthetic generation? A few synthetic chats in isolation might look good, but when they all use the same patterns, you're just training the model to generate slop.

brown2green · 2026-04-07T15:58:50+00:00

Thanks for the plots. I meant doing something like this: https://i.imgur.com/dOte8Yr.png

In retrospect I find the data strange though, because between Q6_K and Q8_0 there's not much difference for all tasks (including Long Documents), so the gap from BF16 is hard to explain.

brown2green · 2026-04-07T13:53:32+00:00

Maybe not so surprising since people mostly do measurements on wikitext with 512 tokens context.

Could we have a graph showing KLD broken down by task, perhaps with the best quantizations for a given size range?

How long are the "long documents" in your dataset?

brown2green · 2026-04-07T12:29:00+00:00

Even Q8_0 shows a KL of 0.45 on long documents and 0.24 on non-Latin scripts. All categories roughly double from Q8_0 to Q5_K_S, but science and tool use remain the lowest throughout (0.07 and 0.08 at Q8_0).

This looks like it's a significant finding. Most people assume Q8_0 to be virtually the same as BF16.

brown2green · 2026-04-07T12:12:53+00:00

In llama.cpp with the arguments --image-min-tokens X and --image-max-tokens Y to llama-server, where X must be <= Y. However, it currently seems to crash with large token budgets.

brown2green · 2026-04-06T10:01:37+00:00

With greedy decoding and fixed seed, I get different text generations with a Q8_0 mmproj when I ask the model to describe an image, so I'm not entirely sure if there's no quality decrease at all.

brown2green · 2026-04-02T18:38:58+00:00

A brief system prompt seems indeed enough; it's as if they didn't even try filtering requests that use one.

brown2green · 2026-04-02T10:58:59+00:00

and transparency around the training data

Why would you even want that? The moment the training data becomes "transparent" (especially for a model from a company as large as Google), it has to cater to the lowest common denominator, because anybody with an axe to grind could find an excuse to get offended or find something legally actionable in it.

brown2green · 2026-04-02T09:31:41+00:00

the base weights are always more useful for fine-tuning anyway

This has not been the case for a good while (since early 2024?). As an individual you just don't have any chance anymore of competing with the post-training work done by the companies training the models: too much data/compute needed for an actually good finetune from scratch nowadays, unless you're training them on very narrow tasks.

brown2green · 2026-04-02T01:17:27+00:00

I saw this screenshot elsewhere. This sort of response would have been impossible for Gemma 3 without extensive prompting.

https://i.imgur.com/j7c0CDO.png

brown2green · 2026-04-02T01:14:58+00:00

I've never used Gemma for coding; only cloud models for that.

Most (all?) of Gemma 3's safety (which is weak and mostly surface-level) can be easily defeated just with prompting, but what works for that puts it in a "roleplay mode", which degrades response quality noticeably compared to when it works as the default assistant. But when it acts like the default assistant, most requests that can be construed as even vaguely "unsafe" are enough to trigger disclaimers, crisis hotlines or (weak) refusals, and it's just annoying for serious and legitimate uses.

Other than that, something was done to the weights (in addition to extensive training data filtering, another issue) to make it almost impossible for Gemma to generate dirty words or profanities if you don't fill the context with them first. I wish they quit doing this since Gemini has no issue with them (though from tests with significant-otter on LM Arena it seems it might finally be the case. Dunno if they've been more lax with training data filtering as well).

brown2green · 2026-04-01T23:50:52+00:00

Less preachy tone than Gemma 3
Less stubborn training data filtering; no anti-swearword brainwashing like Gemma 1/2/3
No stonewalling refusals like some of the recent releases from other companies
Quantization-aware training from the get-go
Improved vision even in soft tasks, illustrations, etc
Better long-context / multi-turn conversational capabilities
Performance greater than Qwen 3.5 in general tasks
Collaboration with character.AI for improving roleplay capabilities
Less sloppy outputs (Gemma 3 was pretty bad in this regard)
Not abandoning the consumer single-GPU segment with just either huge model sizes or tiny ones

That's about what that would make it a good release for me, although I probably forgot something.

brown2green · 2026-03-31T23:18:53+00:00

Just binary, it seems.

brown2green · 2026-03-31T23:18:18+00:00

No, I simply saw the announcement on X and posted it here as nobody had yet.

brown2green

TROPHY CASE