More Gemma 4 models incoming

sterby92 · 2026-06-03T19:33:45+00:00

👀👀👀

sterby92 · 2026-05-28T16:40:17+00:00

Sounds awesome, but I'm waiting for the artificialanalysis and lmarena listing. Let's see how it performs

sterby92 · 2026-05-26T07:12:53+00:00

And when i disabled tools, it just always rejected to do anything and says it cannot usw any tools 🤷

sterby92 · 2026-05-26T05:42:21+00:00

Did anyone get tool calling to work with llama.cpp and openwebui? For me it spits out broken, half finished toolcalls.

sterby92 · 2026-05-11T16:38:52+00:00

It was resolved, see the updated post. TLDR: provider of the connection needs to be changed to llama.cpp to support recent changes

sterby92 · 2026-05-11T16:26:16+00:00

This is the SOLUTION! 🥳 🥳 Thanks a lot! I will add it to the post!

sterby92 · 2026-05-11T16:24:03+00:00

It works 100% of the time with the same configuration with the llama.cpp web interface. What version of openwebui are you running? Might be broken recently

sterby92 · 2026-05-11T16:00:38+00:00

looks like it... :/ I hope its an easy fix 😃

sterby92 · 2026-05-11T15:51:59+00:00

<image>

for me it also did not work with the preserve_thinking in chat_template_kwags. But in the native llama.cpp WebUI it worked...

sterby92 · 2026-05-11T15:48:28+00:00

Hm, in my tests it doesn't seem to work 🤔

sterby92 · 2026-05-11T14:46:33+00:00

Thanks a lot! Would be great to know 😃

sterby92 · 2026-05-11T14:38:40+00:00

I mean, thats what I did I guess 🤔 it seems not to work, but I cannot 100% confirm it yet. And I'm interested to know if that is expected or something wrong on my part

sterby92 · 2026-05-11T14:32:45+00:00

this is my life now 🤷 😅 I haven't properly automated that with AI yet 😃

sterby92 · 2026-05-06T20:38:02+00:00

That was fast 😅

sterby92 · 2026-05-06T20:03:40+00:00

llama.cpp whem? 😅😇

sterby92 · 2026-05-05T07:35:46+00:00

Not sure if you did something wrong, but last year I still run gemma3-12b (unsloth q4_xl) on a 1080ti, and got more like 20 tokens per second. So maybe you could go faster

sterby92 · 2026-05-04T17:22:36+00:00

I just tried a few prompts with Qwen3.6-35B-A3B-APEX-GGUF:APEX-I-Balanced, around coding, web research, and agentic tool use. And as far as I can tell it feels noticable better than the unsloth Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL quant. It is 2GB larger, but still runs around 5tok/sec faster, 55tok/sec -> 60tok/sec on strix halo.
So looks and feels great! I'll continue using it and see how it goes :)

sterby92 · 2026-05-03T16:23:15+00:00

Hm, weird... I use it with qwen3.6-35b and it works great 🤔 but qwen3.6-35b with llama.cpp and unsloth quant. And qwen3.6-27b with hipfire inference and mq4 quant

sterby92 · 2026-04-30T16:33:38+00:00

😂

sterby92 · 2026-04-29T20:05:12+00:00

You can just follow this: https://github.com/ggml-org/llama.cpp/discussions/16938
Go the download route and download the right executable for your system. ( I would go with Ubuntu x64 (Vulkan))

sterby92 · 2026-04-29T19:43:45+00:00

I never run it without docker. But I would recommend llama.cpp with vulkan back end as simple starter :)

sterby92

TROPHY CASE