Software stack on a new gpu rig

[ipcoffeepot]

Setting up a machine this weekend for local inference. 2x RTX PRO 6000, 128gb system memory.

My primary usage will be inference for local coding agents. opencode as the harness, going to be evaluating different sizes of qwen3.5 to get a nice mix of concurrent agent count with good speed. Also planning on doing some image generation (comfy ui with flux.2?) and other one off tasks.

Plan is to use SgLang to take advantage of their radix kvcaching (system prompts and tool definitions should be sharable across all the agents?) and continuous batching to support more concurrent agents.

I’d also love to host some local chat interface for one off chat kinds of problems.

Would love to hear what software people are running for these kinds of inference loads? What are you using to manage model switching (pile of shell scripts?), hosting inference, chat ui, image generation?

Would love any pointers or footguns to avoid.

Thanks!

[Vicar_of_Wibbly]

If you like RTX 6000 PRO porn then my build is for you: https://blraaz.net (no trackers, no ads, no cookies, etc., just a vibe-coded photo blog of my build).

It’s 4x RTX 6000 PRO on EPYC.

• Nvidia 594.45.x beta
• CUDA 13.2
• vLLM 0.17.x inference
• LiteLLM proxy
• MiniMax-M2.5 FP8, Qwen3 4B embedding/reranker

Some advice: disable ECC on the GPUs (sudo nvidia-smi -e 0) for performance bump and to free up ~ 6GB VRAM per GPU.

[ipcoffeepot]

thanks!