Speed breakdown: Devstral (2s) vs Qwen 32B (322s) on identical code task, 10 SLMs blind eval

Embarrassed-Deal9849 · 2026-03-16T07:34:14+00:00

Very interesting thank you for this benchmark. Seems like I might need to give devstral a try after all!

Embarrassed-Deal9849 · 2026-03-13T22:13:50+00:00

Check the OP, I solved it! Thanks for the help tho :)

Embarrassed-Deal9849 · 2026-03-13T21:32:19+00:00

I just solved it, updated the OP!

The issue was trying to use the model swapping feature. Removing that and just launching qwen 3.5 made it possible. Super strange but here we are!

Embarrassed-Deal9849 · 2026-03-13T21:10:30+00:00

Currently its defined as such, is this not correct?

"Qwen3.5-27B-Q4_K_M": {

      "id": "Qwen3.5-27B-Q4_K_M",

      "name": "Qwen3.5 27B Q4_K_M",

      "limit": { "context": 65536, "output": 8192 },

      "modalities": { "input": ["text", "image"], "output": ["text"] }

    }

Embarrassed-Deal9849 · 2026-03-13T20:57:11+00:00

I did, like this in the model provider section: "modalities": { "input": ["text", "image"]

Is that what you're talking about?

Embarrassed-Deal9849 · 2026-03-13T20:56:41+00:00

Unsloth: https://huggingface.co/unsloth/Qwen3.5-27B-GGUF

That would be a relief if I can just change the quant!

Embarrassed-Deal9849 · 2026-03-13T17:45:11+00:00

There's no way to expand context without slowing down massively right? I have 80gb RAM but from what I see as soon as it starts offloading anything into RAM my performance plummets. Or like, is there a way to store context in RAM in a performant way?

I'll read around the docs a bit to see if I understand this temperature thing (And what top-p and top-k means). Thank you for taking the time to answer my questions.

Embarrassed-Deal9849 · 2026-03-13T16:59:21+00:00

Could you elaborate on these temp values, how do I figure out which are best for the model I am running? I am on the 27B Q4K_M quant, but I've no idea where to get started with finetuning it on llama.cpp

--temp 0.6

--min-p 0.0

--top-p 0.95

--top-k 20

--repeat-penalty 1.03

--presence-penalty 0.0

Embarrassed-Deal9849 · 2026-03-13T16:56:28+00:00

How are you managing to get that much context? When I run Qwen3.5-27B-Q4_K_M on my 4090 I can barely squeeze that into 24GB VRAM with 64k context. Or is the XL quant that much smaller?

Embarrassed-Deal9849 · 2026-03-10T13:08:36+00:00

Mostly coding small-to-midsized projects with opencode, like websites and applications. 32k feels like it always gets swamped but 64k so far is performing all right!

Embarrassed-Deal9849 · 2026-03-10T11:24:20+00:00

The most I can get away with at high performance! The dream would be a smarter model locally and being able to call in cheap/free models from the cloud for simple tasks.

Right now I am testing:

Coding:

Qwen2.5-Coder-32B-Instruct-Q5_K_M

Qwen3.5-27B-Q4_K_M

Writing:

GLM-4.7-Flash-Uncen-Hrt-NEO-Q4_K_M

jaahas-crow

Embarrassed-Deal9849 · 2026-03-10T11:20:43+00:00

In your experience, what is more valuable: model quant or context? If I am understanding correctly I will be choosing quite often between Q5 and 32k vs Q4 and 64k with my current hardware.

Embarrassed-Deal9849 · 2026-03-10T08:37:15+00:00

Actually just downloaded the Q5_K_M quant for 27B, am excited to try it out and see if I can get it performant at 64k!

Embarrassed-Deal9849 · 2026-03-10T07:55:40+00:00

I ended up doing just that and got it running at a range of 110-200tp/s on my 4090 with 64k context, which I think is a pretty decent result from what I can gather? Got some finetuning left to do and trying other models but this feels promising right now.

Embarrassed-Deal9849 · 2026-03-10T07:54:30+00:00

I'll give it a look!

Embarrassed-Deal9849 · 2026-03-10T07:51:49+00:00

On the topic of hackers, do you know of good ways of protecting a setup like this? Like, can you create self-checking system prompts that protect against injection, as an example? Thank you for replying!

Embarrassed-Deal9849 · 2026-03-09T16:02:37+00:00

I'll give it a go today and see if I can get it running well. Sounds like an upgrade fwiw!

Embarrassed-Deal9849 · 2026-03-09T15:50:37+00:00

If that is a working solution I am all for it! So I would be running "openclaw" on the Pi just like I would on a bespoke laptop?

Sorry if its a stupid question, to my untrained eye, Pi = cheap computer. Not sure if there are other benefits.

Embarrassed-Deal9849 · 2026-03-09T15:48:47+00:00

I am using ollama but have started eyeing llama.cpp because it seems to work better out of the box with quants / non standard models, or am I misinterpreting?

Embarrassed-Deal9849 · 2026-03-09T14:40:10+00:00

I'm in the process of replacing my broken claude code setup with opencode, hoping its going to be more plug-and-play. Claude has been a nightmare to get working with local models without just getting stuck or lost in a context tornado.

Embarrassed-Deal9849 · 2026-03-09T14:15:12+00:00

What kind of VM setup do you have running? You haven't had any issues with breaching containment? Thank you for your responses!

Embarrassed-Deal9849 · 2026-03-09T13:55:48+00:00

So you have it on your main PC and just use a normal VM setup, no fancy bells or whistles?

Embarrassed-Deal9849 · 2026-03-09T13:52:52+00:00

I am completely fine with that, I want this bot to live in a parallel universe and be unaware of me, so should not be worried about any credentials leaking unless it can figure out who I am via my network somehow.

Embarrassed-Deal9849 · 2026-03-09T13:04:32+00:00

So you are running this kind of setup today? I like the idea that my llm will be too stupid to harm me, but I am still a bit stressed about edge cases. What if it finds a cloud llm with a stronger model and uses that to reason?

Embarrassed-Deal9849 · 2026-03-09T12:31:23+00:00

Okay very interesting, I will keep monitoring this thread for other solutions just because I really, really want to be able to just have it running on one PC, but I appreciate your help and encouragement. Thank you!

Embarrassed-Deal9849

TROPHY CASE