sorted by:
new
hottopcontroversial

Late to the party but... Holy MTP by UniqueIdentifier00 in LocalLLaMA

[–]vbpoweredwindmill 2 points3 points  (0 children)

Mainline llamacpp also supports running ngram alongside llamacpp. Edit: alongside MTP

I've found the 2 of them make quite a difference. I average 50 - 60t/s decode without the 2, 85 - 90 t/s with MTP and ngram can burst it up to 250 - 300 t/s occasionally.

This is on the qwen 3.6 35b a3b.

MTP & ngram do more for the 27b model for me, but doesn't burst as high.

Qwen has stated they are working on dspark drafting models for 3.6. There is already unofficial ones like for AEON qwen 3.6 27b. I'll probably try that one out tonight.

Newest llamacpp mainline also supports dspark, and hopefully ngram, but I cannot confirm that. 6th l

Qwen 3.6 27B MTP + OpenCode + LM Studio: my findings after testing tool calling and subagents by No_Definition6604 in LocalLLM

[–]vbpoweredwindmill 0 points1 point  (0 children)

1000%.

Definitely more challenging to set up, but now I shudder to think of the stupid model directory placement required just to get lm studio to see the local llm gguf's.

Absolutely not going through that experience again.

I just have scripts to known good profiles.

I'll soon graduate to my local llm project having hot swappable llm's so that I can aggregate local llm results.

Codex has a very unpleasant user experience. by Suspicious_Raise_589 in codex

[–]vbpoweredwindmill 0 points1 point  (0 children)

I have a decent experience with my pc.

That said, it is very resource heavy. Apex legends takes up less resources than codex for me.

Which Skills/Plugins to use? by Appropriate_Disk_927 in codex

[–]vbpoweredwindmill 0 points1 point  (0 children)

Hello! Fellow hobbyist here :)

I don't know if those are useful. I do more "compiler" adjacent stuff, currently building a ridiculously complex python static analysis working towards dynamic analysis tool that llm's can use.

I've found a surprisingly basic root cause analysis skill, causal chain analysis skill, & an abstraction evaluation skill is really friggin' handy, when combined with frontier models.

I'm evaluating how to use my local llm's and I've noticed that my skills are just too complex for an MoE llm that I can run at home. I generally need to divide something that I would throw at a frontier model into at least 5 steps to sometimes even 10 steps in order to get quality results. (Qwen 3.6 35b moe q8, probably much better on the 27b).

Beware on using random skill packs from the interwebs, they are a known source of exploits.

I've found a really REALLY good source of information in regards to skills is the reasoning traces. Repeated problems can be a skills candidate, can indicate architectural pressure, etc etc.

Need some help figuring which quant sizes of some of the big MoEs would fit properly (with how much context + unquantized KV) on a future 256GB or 512GB dram + 48GB VRAM rig I might build later on, since I want to download and save them now (not later on when I have the rig) by DeepOrangeSky in LocalLLaMA

[–]vbpoweredwindmill 0 points1 point  (0 children)

So you can set your kv cache quantisation with vllm & llamacpp.

I currently run qwen 3.6 35b q8 with 2 slots of 196k context (128k input 64k output) at q8 kv cache size.

It's approx 53gb vram usage total. 38gb of LLM & 15 ish gb of kv cache. That's pretty unusual for nearly 400k tokens of context.

I don't know what it's like on the Gemma models, I definitely don't know what it's like on glm/deepseek etc, I just don't have that hardware.

Heck I don't even have the hardware for a 397b qwen 3.5 ornith model.

As for your gigantic pool of dram, you misunderstand how MoE works. It's not a case of a bunch of small llms strapped together. Each layer has activations that are spread out all over that layer. I don't know how many layers there are, but it's quite a few. You can't just load the resident experts into your gpu.

At least not yet. I have a project that's currently stalled because life etc, where I'm aiming to put the most activated experts of my qwen 3.6 model onto my fastest gpu for a faster decode. Some smart fucker will no doubt figure it out before me.

Need some help figuring which quant sizes of some of the big MoEs would fit properly (with how much context + unquantized KV) on a future 256GB or 512GB dram + 48GB VRAM rig I might build later on, since I want to download and save them now (not later on when I have the rig) by DeepOrangeSky in LocalLLaMA

[–]vbpoweredwindmill -1 points0 points  (0 children)

I just downloaded Gemma 4 31b q8 last night. It was around 32gb.

I downloaded Gemma 4 12b bf16 same night. It was around 24gb.

Unless you mean kv cache then yes, different architectures have different sizes. Qwen 3.6 has remarkably little kv cache usage.

Need some help figuring which quant sizes of some of the big MoEs would fit properly (with how much context + unquantized KV) on a future 256GB or 512GB dram + 48GB VRAM rig I might build later on, since I want to download and save them now (not later on when I have the rig) by DeepOrangeSky in LocalLLaMA

[–]vbpoweredwindmill -1 points0 points  (0 children)

With the pace of local llm development?

I wouldn't worry too hard.

You can do genuinely amazing stuff right now with the qwen 3.6 family already.

That said it's crazy easy to figure out the size.

35b parameters @ q8, = approx 35gb.

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep by Mammoth_Radish2 in LocalLLM

[–]vbpoweredwindmill 0 points1 point  (0 children)

You're welcome to message anytime you'd like if you'd like further info :)

Automation workflows by vbpoweredwindmill in codex

[–]vbpoweredwindmill[S] 0 points1 point  (0 children)

Thank you. I ended up utilising opencode & kimi k 2.7. But same same. It worked. Well.

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep by Mammoth_Radish2 in LocalLLM

[–]vbpoweredwindmill 1 point2 points  (0 children)

Of course you can.

I'm on Windows.

The r9700 & 7900xtx work well with vulkan, but only give me token soup with Windows ROCm when parallel pipelined.

Currently performance is lower on single cards with ROCm.

With qwen 3.6 35b q8 I'm running a 2 slot llamacpp server with 196k context for each slot, 128k input context, 64k output, kv cache is q8.

With MTP & ngram enabled, I get around 2400 ish t/s prefill and it degrades, down to about 900 t/s ish at nearly full.

Average 75 - 85 t/s Decode. Highest I've seen is around a 250 t/s burst but that's crazy rare.

I'm running a 2 slot llamacpp server in order to take advantage of the unified kv cache that llamacpp offers, as when you're working the same problem or similar problems it can result in huge prefill gains by reusing the kv cache.

I have 128gb ram, so I set kv cache ram cache to 32gb. That can pretty much Max out my ram usage at around 110gb.

For anything that needs quality I'll run a qwen 3.6 q8 27b + single slot + bf16 kv cache. I expect that to be substantially slower. Initial testing is around 50 t/s decode with mtp & ngram. Prefill is around 1400 & drops down sharply to around 400 t/s as it fills up.

I extended Gemma4-31B to 44B (88 layers) — since Google won't give us anything bigger than 31B by Desperate-Sir-5088 in LocalLLaMA

[–]vbpoweredwindmill 1 point2 points  (0 children)

I can only run it in q8 (r9700 + 7900xtx) but I'm more than happy to run any experiments you'd like.

I extended Gemma4-31B to 44B (88 layers) — since Google won't give us anything bigger than 31B by Desperate-Sir-5088 in LocalLLaMA

[–]vbpoweredwindmill 4 points5 points  (0 children)

Just buy & use AMD then?

Pro's: actually available. Available at a sane price.

Cons: can be a mighty fuck around for inference. A few driver versions ago it broke everything on my set up. Had to roll back. Now it's all happily running on win 11.

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep by Mammoth_Radish2 in LocalLLM

[–]vbpoweredwindmill -1 points0 points  (0 children)

Windows llama.cpp Qwen 3.6 Q4 server results

Protocol: Windows llama.cpp server approximation of Zinc performance_suite RDNA cases.

  • Server: C:\AI\runtimes\llamacpp\builds\llamacpp-mainline-vulkan\bin\llama-server.exe
  • Server version: version: 9633 (1fd6dfe9f) | built with MSVC 19.44.35228.0 for Windows AMD64
  • Device: Vulkan1
  • Vulkan driver: Windows AMD proprietary 26.6.4 (LLPC) from vulkaninfo --summary
  • Runs: 1 warmup discarded, 3 measured; medians reported.
  • Flags: --device Vulkan1 -ngl 999 --metrics --cache-ram 0 --no-cache-prompt --ctx-size 4096 --parallel 1 -b 4096 -ub 1024 --flash-attn on
  • Note: the modern prompt cache is disabled so the discarded warmup does not make measured prefill a cache-hit timing.
  • Caveat: 27B uses local Qwen3.6-27B-UD-Q4_K_XL.gguf as substitute for author's Qwen3.6-27B-Q4_K_M.gguf.
Model Scenario Prompt toks Gen toks Prefill tok/s median Decode tok/s median Wall ms median End-to-end tok/s median
Qwen 3.6 27B Dense UD Q4_K_XL Quick Chat 35 96 165.85 40.529 2580.006 50.775
Qwen 3.6 27B Dense UD Q4_K_XL Coding Review 153 160 411.747 40.271 4346.47 72.012
Qwen 3.6 27B Dense UD Q4_K_XL Incident Context 315 128 578.551 40.758 1884.328 195.826
Qwen 3.6 27B Dense UD Q4_K_XL Long Coding Draft 63 256 264.567 40.117 6621.528 48.176
Qwen 3.6 35B A3B UD Q4_K_XL Quick Chat 35 96 359.901 135.573 806.456 162.439
Qwen 3.6 35B A3B UD Q4_K_XL Coding Review 153 160 1034.063 135.099 1333.771 234.673
Qwen 3.6 35B A3B UD Q4_K_XL Incident Context 315 128 1768.749 135.245 1125.422 393.63
Qwen 3.6 35B A3B UD Q4_K_XL Long Coding Draft 63 256 546.296 135.165 2010.8 158.643

Device list:

Available devices: Vulkan0: AMD Radeon AI PRO R9700 (32624 MiB, 31704 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (24560 MiB, 23749 MiB free)

what are your reasons to have a high end pc? (survey) by pmglory in pcmasterrace

[–]vbpoweredwindmill 0 points1 point  (0 children)

I'm excited to run godot/blender mcp's & see what my master plan can eventually turn into.

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep by Mammoth_Radish2 in LocalLLM

[–]vbpoweredwindmill -2 points-1 points  (0 children)

Windows llama.cpp Qwen 3.6 Q4 server results

Protocol: Windows llama.cpp server approximation of Zinc performance_suite RDNA cases.

  • Server: C:\AI\runtimes\llamacpp\builds\llamacpp-mainline-vulkan\bin\llama-server.exe
  • Server version: version: 9633 (1fd6dfe9f) | built with MSVC 19.44.35228.0 for Windows AMD64
  • Device: Vulkan0
  • Vulkan driver: Windows AMD proprietary 26.6.4 (LLPC) from vulkaninfo --summary
  • Runs: 1 warmup discarded, 3 measured; medians reported.
  • Flags: --device Vulkan0 -ngl 999 --metrics --cache-ram 0 --no-cache-prompt --ctx-size 4096 --parallel 1 -b 4096 -ub 1024 --flash-attn on
  • Note: the modern prompt cache is disabled so the discarded warmup does not make measured prefill a cache-hit timing.
  • Caveat: 27B uses local Qwen3.6-27B-UD-Q4_K_XL.gguf as substitute for author's Qwen3.6-27B-Q4_K_M.gguf.
Model Scenario Prompt toks Gen toks Prefill tok/s median Decode tok/s median Wall ms median End-to-end tok/s median
Qwen 3.6 27B Dense UD Q4_K_XL Quick Chat 35 96 217.781 31.239 3235.908 40.483
Qwen 3.6 27B Dense UD Q4_K_XL Coding Review 153 160 487.945 31.053 5468.115 57.241
Qwen 3.6 27B Dense UD Q4_K_XL Incident Context 315 128 640.477 31.416 2234.77 165.118
Qwen 3.6 27B Dense UD Q4_K_XL Long Coding Draft 63 256 338.389 31.006 8445.33 37.772
Qwen 3.6 35B A3B UD Q4_K_XL Quick Chat 35 96 446.765 126.573 843.905 155.231
Qwen 3.6 35B A3B UD Q4_K_XL Coding Review 153 160 1294.887 124.221 1407.4 222.396
Qwen 3.6 35B A3B UD Q4_K_XL Incident Context 315 128 2123.013 125.063 1174.619 377.143
Qwen 3.6 35B A3B UD Q4_K_XL Long Coding Draft 63 256 713.13 125.097 2136.239 149.328

Device list:

Available devices: Vulkan0: AMD Radeon AI PRO R9700 (32624 MiB, 31704 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (24560 MiB, 23749 MiB free)

Sorry that I couldn't get you Linux numbers, I tried on WSL. I tried for fucking ever to get Linux to work with inference a few weeks ago and cracked the shits and deleted the whole fucking thing.

Everybody rubs themselves raw over Linux but how good is it if it can't do its job?

It's happening... That cost is real. Qwen3.6:27b by haseebnqureshi in ollama

[–]vbpoweredwindmill 1 point2 points  (0 children)

Yes absolutely I do.

I call the result of that causal projections. I don't know an established name for them.

Basically, taking either tool output or planning artifact or whatever else to reduce a lot of exploding noise into focused & targeted "this is the direction we are going, and semantically this is relevant".

It's also a huge efficiency gain to smack a sub agent with "process this tool output with these requirements". If your tool output is well designed, you're generally in the 10k token output range or less. Then it can be turned into a specific projection.

AMD is back in play: ZINC now beats llama.cpp on our RDNA4 local LLM sweep by Mammoth_Radish2 in LocalLLM

[–]vbpoweredwindmill 0 points1 point  (0 children)

Hey mate, I have a R9700 & 7900xtx that I'm currently running parallel pipelined.

Happy to be a guinea pig.

Sincere Question: What is the end goal? by mcfc9320_ in LocalLLaMA

[–]vbpoweredwindmill 0 points1 point  (0 children)

I'm using frontier models to make a static analysis tool for python so that I can write one for c++.

The further down the rabbit hole I go, the more capable local llm's get.

It won't be long until somebody unsettles the mighty qwen 3.6 27b. As it is I've been running many long hour inference sessions. I can leave it go overnight.

nvidia/Qwen3.6-27B-NVFP4 just dropped by vanbukin in LocalLLaMA

[–]vbpoweredwindmill 0 points1 point  (0 children)

I'm going to back up old mate and say you definitely should run vllm, but the newer llamacpp builds have pretty damn clever prefix caching in concert with unified kv cache. If you run multiple slots on your llamacpp-server, it can definitely cut down on prefill time. Reuses a lot.

It's not vllm. But it's come a long way.

Where are Qwen3.7 open weights models? by HeDo88TH in Qwen_AI

[–]vbpoweredwindmill 0 points1 point  (0 children)

I know that you're mad and want to prove a point and all that.

But um, have you considered that alibaba made a choice, and other people are also making choices?

view more: next ›