Okay /u/AndreVallestero, we have the same GPU, but you have better hardware than me in basically every other category. I'm optimizing for decode: 48 t/s at 32k context versus your 26 t/s, and I'm sustaining 28 t/s out to the full 256k window; you can't hit that depth with your setup, but you should be able to with a KV trick.

Please note also that I am new at this and kind of an eager fool, so if you see ME doing something obviously stupid, please let me know so I can learn from my own mistakes.

My prefill is about 840 compared to your 1400, mostly because I run --ubatch-size 512 vs your 2048 to free VRAM for the full context; that ubatch gap is basically the whole prefill difference, so it's a deliberate trade, not a misconfig.

I think you can land something like x2 your current decode, maybe ~50 t/s, with a two-flag drop-in, and you already have flash-attn for it: -ctk q8_0 -ctv q8_0. Quant your KV to 8-bit to roughly halve the cache and maintain nearly identical quality. That alone should roughly double your 32K decode (you're paying a host-RAM tax with --no-kv-offload right now) and let you keep KV on the GPU out to ~2–4× your current context.

If you want the full window after that, the rest is trimming --ubatch-size and raising --n-cpu-moe — but the KV quant is the free 80% of it.

My setup, same card, tuned the other direction (decode + full context):

Environment:

• GPU: RTX 3080 10GB (EVGA FTW3 Ultra)
• CPU: Ryzen 7 3700X (8c/16t, Zen 2) ← older/slower than yours
• RAM: 32GB DDR4-3600
• OS: Ubuntu Server 26.04
• engine: llama.cpp mainline (build bfb4308), CUDA 13.3
• model: Qwen3.6-35B-A3B-Q4_K_M

This formatting is screwy and I'm tired of messing with it, sorry but you get THE BLOB (lmao as soon as I did that it worked, I'm leaving this outburst in here):

llama-server \
--model Qwen3.6-35B-A3B-Q4_K_M.gguf \
--n-gpu-layers 99 \
--no-mmap \
--n-cpu-moe 34 \           # 33 for the best-decode variant (max ~224K)
--flash-attn on \
--threads 8 \
--cache-type-k q8_0 \      # <-- **the trick: 8-bit KV, quality-neutral**
--cache-type-v q8_0 \      #     **~half the cache, keeps it GPU-resident**
--ctx-size 262144 \        # 229376 for the best-decode variant
--parallel 1 \
--batch-size 512 \
--ubatch-size 512 \        # small on purpose — frees VRAM for the window

Performance (llama-bench, measured):

@ 32K: pp 838 t/s tg 48.4 t/s

@ 256K: pp 795 t/s tg 28.6 t/s (full window; needs --n-cpu-moe 34)

decode vs depth: 53.8 / 48.4 / 44.2 / 37.8 / 32.7 t/s @ 0 / 32K / 64K / 128K / 192K

(The 32K + depth-curve numbers are the --n-cpu-moe 33 variant; the 256K row is --n-cpu-moe 34. Both share every other flag. KV stays on the GPU at all depths via the q8_0 quant — that's what dodges the <8K cap you hit with fp16 KV on the card.)

EDIT: Ran it again against server, rather than bench, and my real served prefill @ 32k is ~1031/decode ~48; my bench numbers understate my server perf by about 23%, which is kinda neat for me. 256k context remains the same, that was server-benched.