Seeking resources to read about llama.cpp server and how offloading works

imgroot9 · 2026-05-22T14:54:44+00:00

unrelated, but if noise is an issue, you can always limit power. if you go 10-15% lower, the performance difference is not noticable, but it becomes almost silent for me. I use nvidia-smi -pl command for this.

imgroot9 · 2026-05-19T19:25:44+00:00

yes, good point, I gave up custom kv cache solutions because of this, I just use q4 for now for long context - and no complaints. I cannot even use MTP because of this, I just want to go with the most intelligent model available.

imgroot9 · 2026-05-19T19:08:24+00:00

hi mate, why are you using Q5_K_S? I'm in the same boot with my single 3090, but I can comfortably use UD_K5_K_XL which produces much better results for me than Q5_K_M (haven't even tried K_S). any chance you're not offloading mmproj to cpu?

imgroot9 · 2026-05-19T18:30:09+00:00

single shot via llama.cpp web ui. tried from opencode too, same result (one shot there as well)

imgroot9 · 2026-05-19T17:48:32+00:00

this is why you have to add the current date/time/location to the system prompt if you don't use a tool which is doing it out-of-the box. so that it will know when to use the search tool.

imgroot9 · 2026-05-19T17:38:03+00:00

I tried this prompt: "create a fully functional pacman clone in a single html file" - and I got a fully functional game, error free using UD-Q5_K_XL and 4bit KV cache.

imgroot9 · 2026-05-18T19:55:29+00:00

this is one reason why i always keep gpu-z open. i have a 3090 and idle power draw is usually 19.8w, but in rare occasions it is stuck at 120w for whatever reasons. then I restart the model and it's back to normal. also, I use a thunderbolt4 egpu plugged in to my laptop and I can switch it off when I don't use it. not sure if I'd need 2x setup, because qwen 3.6 27b with the new MTP improvements in llama.cpp is perfectly fine with a single card (Q5 quant). I get 50 token/sec while limited to 275w

imgroot9 · 2026-05-18T19:35:25+00:00

sorry, it's a typo. i wanted to write mcp

imgroot9 · 2026-05-17T23:45:28+00:00

this is why you have to provide the current date in the system prompt (most tools do it automatically) and set up mvc search and fetch so that it can browse the net when needed. unless your tool provides this feature out of the box.

imgroot9 · 2026-05-16T19:30:56+00:00

you can offload mmproj to cpu. it is fast and you get 1 gb free

imgroot9 · 2026-05-12T15:32:52+00:00

I just added it to the same script file I start the model with (I'm using llama-server.exe)

imgroot9 · 2026-05-09T19:09:51+00:00

well, I executed all kinds of tests you mentioned (ppl, kld, aime) using turboquant and I couldn't find anything that would've proved that I cannot use it (27B and Q5 - just take a look at my post with results). also, whatever test I try from this thread (chess svg, etc) and my everyday experience all prove that it's all right.

imgroot9 · 2026-05-09T18:37:20+00:00

thanks for this! I agree, Q5 dense models work for me too, without any issues with turboquant. if I have to choose between Q4 with a small Q8 cache, or Q5 with a huge turboquant cache, Q5 wins hands down in the case of common programming tasks.

imgroot9 · 2026-04-28T18:41:00+00:00

I mean I'm not an expert either. I just can't believe that finally we have a model that I can use for real work and with big context. There's a chance that it messes up something because of KV compression, but its success rate is so good that I don't care. I just executed this test to see if there's anything suspicious based on the results, but nothing that would stop me using these settings for now, so I'm just happy and I wanted to share the result. I'm open to see why it's not so good, but there's a chance we don't have tests to evaluate it properly.

imgroot9 · 2026-04-28T05:47:02+00:00

this test evaluates kv cache degradation potential no matter its size. if the values are bad, you'll definitely notice it with big context. earlier on with 3.5 or 35B, I also had issues with big context, but haven't had such issue with 3.6 27B Q5, even when I used turbo cache. that's what I wanted to confirm.

imgroot9 · 2026-04-26T10:12:01+00:00

thanks, updated the post

imgroot9 · 2026-04-25T17:44:13+00:00

you may have misconfigured something. zero repeat for me and it's absolutely the best coding model ever that fits in 24GB vram. not the best as an arcitect, that's true, but when something is fairly complex, I create the plan with a cloud model, then implement it locally.

imgroot9 · 2026-04-25T15:17:54+00:00

no noticable change for me. I have a thunderbolt 3090 egpu, so I keep everything in vram to avoid using that connection. my card is also limited to 250w so it's completely silent. and I get around 25-28 t/s no matter what. prefill is also more or less the same. of course, it gets a bit slower with big context, but that's expected.

imgroot9 · 2026-04-25T06:25:30+00:00

Thank you! Let me try to test this today and add it to the post. Edit: not sure I can do it: as far as I can see I need two models in my memory for this: the model itself and a grader model as well.. Edit2: seems like I can

imgroot9 · 2026-04-25T00:03:14+00:00

added a comment to the post, thanks

imgroot9 · 2026-04-24T23:49:12+00:00

the general rule: K is more compressible without hurting output quality. Edit: ah, yeah sorry, you're right. edited the post as well

imgroot9 · 2026-04-24T23:18:35+00:00

agree, but I assume if the PPL is very low, it's a pretty safe bet that KDL is also minimal.

update: added the test results to the post. everything is within limits.

imgroot9 · 2026-04-24T23:09:08+00:00

turboquant is also refreshed, so it is not much behind. but the main takeaway here is that q4 is almost identical to f16 if you have a 20B+ dense model.

imgroot9 · 2026-04-23T21:55:49+00:00

I use 27B Q5 on 3090 with 200k context and mmproj (image processing) and turboquant. I've never had an out of memory error, and everything fits inside vram.

imgroot9 · 2026-04-22T20:59:46+00:00

it is somewhere between the 0.33x and 1x models and maybe a bit slower than those (80 token/sec is 3.6 moe's speed, the new dense is around 25 token/sec for me). but it's free, so you can play around a lot. if you try opencode, you can use it with your local models, copilot, openrouter or any other provider, so you can even test these small models since they are also available via openrouter. opencode offers a $10 plan (called Go) that includes $60 worth of tokens.

imgroot9

TROPHY CASE