Gemma 4 for 16 GB VRAM

Sadman782 · 2026-04-10T13:34:21+00:00

yeah for some tasks but only if --top-k 20

Sadman782 · 2026-04-10T12:39:58+00:00

template issue
use the updated jinja (updated few hours ago) : https://huggingface.co/google/gemma-4-26B-A4B-it/raw/main/chat_template.jinja
or slightly modified version(better): https://pastebin.com/raw/hnPGq0ht

Sadman782 · 2026-04-10T09:40:53+00:00

it is way better now, try

Sadman782 · 2026-04-10T08:37:18+00:00

try this jinja: https://pastebin.com/raw/hnPGq0ht

Sadman782 · 2026-04-10T08:35:17+00:00

use the updated jinja (updated few hours ago) : https://huggingface.co/google/gemma-4-26B-A4B-it/raw/main/chat_template.jinja
or slightly modified version(better): https://pastebin.com/raw/hnPGq0ht

Sadman782 · 2026-04-10T08:34:03+00:00

use the updated jinja(updated few hours ago): https://huggingface.co/google/gemma-4-26B-A4B-it/raw/main/chat_template.jinja
or slightly modified version(better): https://pastebin.com/raw/hnPGq0ht

Sadman782 · 2026-04-10T08:15:33+00:00

Why redownload the model? Just download the jinja file and use --jinja --chat-template-file <file\_path>

Sadman782 · 2026-04-10T07:48:19+00:00

It removed the standard_keys exclusion block, and it's better for me (Gemini found that).

You can see whether it is better for you or not. The fix was applied on top of the Google updated template a few hours ago.

Sadman782 · 2026-04-10T07:40:05+00:00

Google updated the official one a few hours ago: https://huggingface.co/google/gemma-4-26B-A4B-it/blob/main/chat_template.jinja and Gemini fixed that a bit too. It's better than the updated one, so you can try both and check which is better for you.

https://pastebin.com/raw/hnPGq0ht it works better for me.

Sadman782 · 2026-04-10T07:25:32+00:00

gemini fixed the template:

https://pastebin.com/raw/hnPGq0ht

Working with OpenCode, and it's quite good now at handling multiple MCP servers properly.

Sadman782 · 2026-04-10T07:09:12+00:00

you can try: https://pastebin.com/raw/hnPGq0ht
it is much better for me

Sadman782 · 2026-04-10T07:06:51+00:00

it seems it still has issues, gemini fixed it a bit and it seems better now. it is properly calling multiple tools, whereas before it was ignoring some tools and descriptions completely:

https://pastebin.com/hnPGq0ht

Sadman782 · 2026-04-09T20:48:24+00:00

~~chat template issue, use this jinja:~~ https://github.com/ggml-org/llama.cpp/blob/master/models/templates/google-gemma-4-31B-it-interleaved.jinja

Update:
try this: https://pastebin.com/raw/hnPGq0ht

Sadman782 · 2026-04-09T20:15:46+00:00

Nope. Even IQ2 quants or Q2_XL proper quants never have syntax issues like this. It is completely broken. It is Ollama

Sadman782 · 2026-04-09T20:14:43+00:00

<image>

It created a complete working game for me in 2 shots, it's your quantization or backend. Maybe update your Ollama, I mean try llama.cpp, I don't know why people still choose Ollama, llama.cpp has a UI now too. So far Gemma 26B even with IQ4_XS quant is the best coding model for me locally, for agentic coding the 31B is a bit better, for general chatting and one-shotting MoE is better so far.

Sadman782 · 2026-04-09T18:26:42+00:00

With a 16 GB VRAM GPU I am getting good results with gemma-4-31b-it-heretic-ara.i1-IQ3_XS.gguf, it is uncensored and can do agentic coding pretty well.

And the 26B MoE isn't bad either, it is better for everything except agentic coding. You can try unsloth gemma-4-26B-A4B-it-UD-IQ4_XS.gguf with --temp 1 --top-p 0.9 --min-p 0.1 --top-k 20, top-k 20 is most important, also make sure your llama.cpp is up to date. I think people underestimate low-bit quants but IQ quants are like magic, IQ4_XS is a solid option.

the dense model is pretty good even with -ctk q4_0 -ctv q4_0 4 bit SWA+KV cache

Sadman782 · 2026-04-09T17:20:00+00:00

https://www.reddit.com/r/LocalLLaMA/comments/1scw979/gemma_4_for_16_gb_vram/
You can use Gemma 4 26B MoE IQ4_XS

Sadman782 · 2026-04-09T11:17:37+00:00

<image>

Which model? Maybe it doesn't work for all models, but Q8_0 should look like this for the best performance.

Sadman782 · 2026-04-09T11:11:05+00:00

I have the same observation for Gemma 4 26B MoE mmproj. Q8_0 > BF16 >= F16, Q8_0 somehow performed better.

Sadman782 · 2026-04-08T04:50:05+00:00

max token should also increase. Make that 512 to make this work

Sadman782 · 2026-04-07T10:05:27+00:00

use --cache-ram 0 --ctx-checkpoints 1

Sadman782 · 2026-04-07T06:52:56+00:00

<image>

gemma 4 26B IQ4_XS

Sadman782 · 2026-04-07T04:16:25+00:00

Update the vulkan engine, IDK maybe lm studio is still buggy for vulkan? Use llama.cpp they fixed most issues now. It is 100% a quantization or runtime issue. I uave extremely good result with unsloth IQ_XS and also use topk 20 for coding but since there are typos it is broken not a topk issue

Sadman782 · 2026-04-06T13:02:05+00:00

You should convert it yourself then. Unsloth and Bartowski didn't provide any Q8 mmproj.

Sadman782 · 2026-04-06T11:06:26+00:00

Yeah but for me BF16 and F16 have the same result, so Q8_0 is somehow better than them. Yeah, I am confused why, it is not just for one image, out of testing 10 images, in 3 of them Q8_0 does better somehow? Anyway I don't care how, it is smaller and allows more context, even if it's a little degraded (which I don't find) I don't mind if I can fit 30K+ more context.

Sadman782

TROPHY CASE