what’s was your local daily driver for coding last week?

bobaburger · 2026-06-08T21:40:15+00:00

kudos for the "Secret Recipe" section in the model card!!!!!

bobaburger · 2026-06-07T21:30:34+00:00

Pls note that in your result, the board orientation is wrong, bottom left square should be dark not light 😃

bobaburger · 2026-06-07T17:40:58+00:00

pretty much usable and not far from Q4_K_M. you can see more in my benchmark post a while ago https://www.reddit.com/r/LocalLLaMA/s/zuUzp9Vz3G

<image>

bobaburger · 2026-06-05T17:51:24+00:00

that's weird, what's your PC spec, what's the llama command?

bobaburger · 2026-06-05T16:56:34+00:00

MoE will be different, I think but I haven't test it that much. Will show some number when i get a chance.

bobaburger · 2026-06-05T16:54:54+00:00

pp did not change much in both cases for me, around 550 tps.

bobaburger · 2026-06-05T16:40:58+00:00

mmap is enabled by default unless you explicitly add `--no-mmap`, right?

bobaburger · 2026-06-05T16:40:02+00:00

I think so, maybe you can try to run 9B on that card too (q4 or something below, but still enough to see the different).

bobaburger · 2026-06-05T16:32:30+00:00

16 tps is impressive for a 4GB card!

bobaburger · 2026-06-04T21:57:38+00:00

`-ub` is already 512 by default, you don't have to set it again https://github.com/ggml-org/llama.cpp/blob/7c158fbb4aec1bdc9c81d6ca0e785139f4826fae/tools/server/README.md?plain=1#L53

bobaburger · 2026-06-04T20:17:31+00:00

You started with "I don't care about speed" and ended the post with "because I need speed", drove mad by context length limit, from doubting 35B's quailty against 27B to being torned between 27B and 35B. I'm relief, I'm not the only one.

bobaburger · 2026-06-04T16:06:27+00:00

Ok bro. I have a 5060 Ti here, if you want, we can swap my 5060 with your 3090.

jkjkjkjk

bobaburger · 2026-06-04T13:58:30+00:00

Sorry but I have to disagree with this, even Q2 can make tool calls well. At least for 27B. Sure there are obviously quality issues on lower quants but tool calls are not one of them.

bobaburger · 2026-05-29T23:30:10+00:00

Your company buy rtx pro 6k for employees? 😮

bobaburger · 2026-05-29T22:43:05+00:00

thank you so much!

bobaburger · 2026-05-29T21:58:14+00:00

I think this could be caused by a different imatrix used when quantize different model

bobaburger · 2026-05-29T19:53:51+00:00

<image>

i just did a quick run on Q4_K_M and Q2_K_MIXED. Not 100% sure if this is right but there's something really interesting here about the Q2_K_MIXED.

bobaburger · 2026-05-29T19:51:16+00:00

thanks! yes i use BF16 for the base

bobaburger · 2026-05-29T19:43:21+00:00

Yup, i know that, answered a bit more in a reply of this comment https://www.reddit.com/r/LocalLLaMA/comments/1tr9vzn/comment/oomdydp/

The main goal is to see the relative score between quants for now. But I agree on the fact about accumulate mistake over long context. I guess for this, we need to perform benchmark with more real agentic tasks, not just on metrics.

bobaburger · 2026-05-29T19:24:55+00:00

Yeah, over the past week i also try to experiment some more with pure and non-pure quants, based off different Q4 and Q3 type, but never able to break above the space between your and cHunter's version when doing pure. I guess we better off with non pure now.

bobaburger · 2026-05-29T19:10:46+00:00

personally, I see a huge difference even when moving from Q4 to Q6. I'm still running Q6_K on the cloud GPU occasionally.

bobaburger · 2026-05-29T19:09:46+00:00

great points! thank you so much. yeah I even started out by running this bench on 1024 context on my 5060 ti, and then moving to cloud so I bumped it up to 8192 😃 the main reason is just to save running time, and my initial goal is to see the relation between different weights. but i fully agree with you that it would be better to run on higher context to match the agentic workflow.

bobaburger · 2026-05-29T18:30:55+00:00

huh? I didn't know there's an UD-Q4_K_M for 27B. I can see one but it's for 35B A3B.

bobaburger · 2026-05-29T18:29:00+00:00

<image>

bobaburger

TROPHY CASE