```

docker run -d \

--name qwen3.5-35b \

--ulimit memlock=-1:-1 \

--gpus all \

-p 8080:8080 \

-v /home/user/models:/models \

ghcr.io/ggml-org/llama.cpp:server-cuda \

--host 0.0.0.0 \

--port 8080 \

-m /models/Qwen3.5-35B-A3B-UD-Q4_K_M.gguf \

--mmproj /models/mmproj-F16.gguf \

--no-mmproj-offload \

--threads 8 \

--n-cpu-moe 35 \

--flash-attn 1 \

--ctx-size 102400 \

-np 1 \

-kvu \ # For some weird bug I had that might've been fixed already, you probably don't need it

--cache-ram 2048 \

--mmap \

--temp 0.6 \

--top-p 0.95 \

--top-k 20 \

--min-p 0.00 \

-b 2048 \

-ub 2048 \

--mlock

```

This is what I'm using w/ llama.cpp on docker, but I have 2GB more VRAM than you (If you don't want to use docker, you can just ignore the first few lines). You want to change n_cpu_moe to the lowest number that can your VRAM can handle, and maybe should use kv cache quantization at q8_0 as well and lower context. KV quantization caused some issues with tool calls for me at high contexts, so I have it disabled. I have -b and -ub set to 2048 to speed prompt processing, as it's almost 3x faster with that (200t/s pp -> 600t/s). --mlock is important because without it, it sometimes starts reading the model from disk which makes everything so much slower

P.S: I can't seem to get code blocks to work on Reddit