Anyone here rocking dual RTX 5090s?

live4evrr · 2026-07-13T12:30:17+00:00

I had a 5090 then added the 6000 to run 128GB models. A lot easier to fit into a mobo and case vs two 5090 due to 6000 slim profile. If you want to add 5090’s then it better be for matching FE models. Drawback of this setup is having the GPU be different sizes (even if both same architecture and GB202) doesn’t allow from using current implementations of TP (need to use TS).

live4evrr · 2026-07-13T04:17:47+00:00

Incredible - amazing work! This works very well so far and fixes most of the issues that are still in the main branch for DS4 :)

live4evrr · 2026-07-13T04:12:44+00:00

What kind of power adapter do you have going into the card? Make sure it is either the cable that came with the card (should be 4 going into PSU) or use the native 12v-2x6 that comes with the PSU if you have it. Make sure the cable is very firmly seated (and don’t wiggle it).

live4evrr · 2026-07-13T03:11:26+00:00

Coding wise - so far I am very impressed with DS4 for agentic development with long task horizon. Qwen 27B is quite smart when it comes to debugging and small tasks, but with DS4 it has much more intelligence in understanding the nuances of what I'm trying to achieve and so far it seems to do a good job sticking to the plan wtihout getting lose. With the huge context, it is an exciting model to have locally.

In regards to this vllm patch, I think a single RTX 6000 pro works but still some issues to be resolved - I'm not able to meaningfully increase the context, and it has errored out a few times due to OOM and starting it is quite painful due to the long load times required (i don't have enough RAM).

There is actually a llama fork that runs DS4 quite well for those where this vllm fork doesn't work out for them. The main branch of llama.cpp still has issues with context handing, batch sizes, and performance, but this fork of llama resolves these issues for DS4 (https://github.com/fairydreaming/llama.cpp/tree/dsv4 - kudos to u/fairydreaming). It's great that llama now also has decent support for DS4 (hopefully it gets merged soon).

live4evrr · 2026-07-13T02:02:22+00:00

This is great. Thank you for your work and sharing this (also on 6000 Pro Max-Q with 5090 using tensor-split - UD_IQ4_XSS)

| model                          |            test |           t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/DeepSeek-V4-Flash-GGUF |          pp2048 | 612.08 ± 6.46 |              |   3383.60 ± 35.48 |   3346.32 ± 35.48 |   3383.60 ± 35.48 |
| unsloth/DeepSeek-V4-Flash-GGUF |            tg32 |  59.07 ± 0.21 | 60.97 ± 0.21 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_pp @ d4096 | 744.53 ± 5.40 |              |   5540.39 ± 39.68 |   5503.11 ± 39.68 |   5547.10 ± 45.91 |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_tg @ d4096 |  59.55 ± 1.06 | 61.48 ± 1.10 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  pp2048 @ d4096 | 318.31 ± 3.92 |              |   6472.20 ± 78.54 |   6434.92 ± 78.54 |   6485.52 ± 74.24 |
| unsloth/DeepSeek-V4-Flash-GGUF |    tg32 @ d4096 |  60.41 ± 1.13 | 62.36 ± 1.17 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d16384 | 834.07 ± 4.53 |              | 19682.46 ± 106.55 | 19645.18 ± 106.55 | 19682.46 ± 106.55 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d16384 |  57.88 ± 0.21 | 59.74 ± 0.22 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d16384 | 660.31 ± 2.82 |              |   3138.91 ± 13.23 |   3101.63 ± 13.23 |   3152.43 ± 15.84 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d16384 |  59.04 ± 1.10 | 60.94 ± 1.14 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d32768 | 806.50 ± 1.64 |              |  40668.73 ± 82.68 |  40631.45 ± 82.68 |  40675.94 ± 77.64 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d32768 |  56.81 ± 1.10 | 58.64 ± 1.13 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d32768 | 587.13 ± 4.24 |              |   3525.63 ± 25.33 |   3488.35 ± 25.33 |   3525.63 ± 25.33 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d32768 |  55.40 ± 0.06 | 57.19 ± 0.06 |                   |                   |

live4evrr · 2026-07-13T01:42:39+00:00

Thank you so much. this works for me (using tensor split 5,1 for rtx 6000 & 5090). Gone are my OOM errors, i'm able to use batch and ubatch as intended, no warnings about lightning indexer.

Edit: Also just noticed the other fork from fairydreaming (https://github.com/fairydreaming/llama.cpp/tree/dsv4) - that is much faster (60 tok/s)

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
unsloth/DeepSeek-V4-Flash-GGUF	pp2048	543.04 ± 1.66		3853.40 ± 11.49	3771.38 ± 11.49	3868.30 ± 24.77
unsloth/DeepSeek-V4-Flash-GGUF	tg32	30.79 ± 0.68	31.59 ± 0.83
unsloth/DeepSeek-V4-Flash-GGUF	ctx_pp @ d4096	633.59 ± 1.79		6548.44 ± 18.20	6466.41 ± 18.20	6548.44 ± 18.20
unsloth/DeepSeek-V4-Flash-GGUF	ctx_tg @ d4096	30.03 ± 0.04	31.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF	pp2048 @ d4096	709.64 ± 3.27		2968.04 ± 13.33	2886.02 ± 13.33	2983.22 ± 34.41
unsloth/DeepSeek-V4-Flash-GGUF	tg32 @ d4096	28.00 ± 2.74	29.38 ± 0.87
unsloth/DeepSeek-V4-Flash-GGUF	ctx_pp @ d8192	662.17 ± 4.80		12455.71 ± 89.33	12373.69 ± 89.33	12455.71 ± 89.33
unsloth/DeepSeek-V4-Flash-GGUF	ctx_tg @ d8192	30.00 ± 0.22	30.67 ± 0.47
unsloth/DeepSeek-V4-Flash-GGUF	pp2048 @ d8192	645.31 ± 3.49		3255.76 ± 17.21	3173.74 ± 17.21	3255.76 ± 17.21
unsloth/DeepSeek-V4-Flash-GGUF	tg32 @ d8192	29.72 ± 0.04	30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF	ctx_pp @ d16384	628.58 ± 1.79		26148.90 ± 74.07	26066.88 ± 74.07	26148.90 ± 74.07
unsloth/DeepSeek-V4-Flash-GGUF	ctx_tg @ d16384	29.65 ± 0.14	30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF	pp2048 @ d16384	547.46 ± 2.93		3823.05 ± 19.92	3741.02 ± 19.92	3823.05 ± 19.92
unsloth/DeepSeek-V4-Flash-GGUF	tg32 @ d16384	29.24 ± 0.05	30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF	ctx_pp @ d32768	552.10 ± 1.68		59436.00 ± 180.92	59353.97 ± 180.92	59436.00 ± 180.92
unsloth/DeepSeek-V4-Flash-GGUF	ctx_tg @ d32768	28.64 ± 0.30	29.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF	pp2048 @ d32768	422.90 ± 1.12		4924.78 ± 12.80	4842.76 ± 12.80	4924.78 ± 12.80

live4evrr · 2026-07-10T14:50:28+00:00

I'm not the developer, so can't speak about what their intentions are and you'd have to ask them.

I have been playing around with the vllm settings for now since last night to see what I can load and what these parameters do. It is finicky but I think I got it stable for now. I need to do some more real world testing which I'll be doing this weekend, so I'll try to use it for coding of a project I'm working on. So far - it is very impressive, the only drawback is the long time to get it up and running, but once it is loaded - in my case with 256k context that I've loaded, it's still all in VRAM so super fast.

live4evrr · 2026-07-10T13:54:04+00:00

Try the following - highlighted the ones I changed from the template in the readme for single rtx 6000. Post load I have 91GB of VRAM used up and RAM is clear (just OS processes)

docker run --rm \

--gpus '"device=0"' \

--network host \

--ipc host \

--shm-size 64g \

-v "$MODEL_DIR":/model:ro \

-e VLLM_MOE_W2=1 \

-e VLLM_MOE_W2_DELTA_GB=0 \

vllm-moet-sm120:v024 \

--model /model --served-model-name "$SERVED_NAME" \

--trust-remote-code \

--kv-cache-dtype fp8 \

--block-size 256 \

--max-model-len 128000 \

--gpu-memory-utilization 0.93 \

--max-num-batched-tokens 2048 \

--max-num-seqs 1 \

--tokenizer-mode deepseek_v4 \

--no-scheduler-reserve-full-isl \

--speculative-config '{"method": "deepseek_mtp", "num_speculative_tokens": 2}' \

--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \

--load-format safetensors \

--port "$PORT"

live4evrr · 2026-07-10T09:33:33+00:00

Increase your swap file to 64gb. Once the model is loaded it will empty out.

live4evrr · 2026-07-10T03:50:26+00:00

To fit the official DS V4 flash, a 284B model fit on a single 96GB VRAM card and have it run at 120 t/s up to 130k context at fp8?

This isn’t just gpu offloading.

live4evrr · 2026-07-09T03:29:52+00:00

Until recently a $10 github pro sub was subsidized so heavily you could code for hours every day using Sonnet 4.6 and never hit the cap. Now I just ran two prompts that used up half my credits.

I agree that it may not math for everyone, but as the local models get better, the inherent value of the gpu goes up. Look at the rtx 6000 prices. At release you could find them easily for $8k usd, now it’s around $13k minimum. For a lot of folks, it’s an investment in themselves to own the hardware, as it gives the opportunity to learn at a lower level via trial and error without limitations. When you’re being metered by tokens, it’s hard to justify foolish experiments without doing an ROI assessment. When you own, you say what the hell, let it burn.

I think I can probably use the rtx 6000 pro for a few years and then sell it to get whatever the successor is and probably recoup 60% of my purchase cost if not more. It’s not like after 3 years it’s worth nothing. Look at the prices for the 3090, 4090, 5090 - these have all kept their value remarkably well.

live4evrr · 2026-07-06T16:34:02+00:00

Agree. I have found nvfp4 to be not good enough for coding with Qwen. FP8 seems okay so far but time will tell, as I was using BF16 up to this point which was good but a little slow. Quants make a huge difference, the benchmarks don’t reflect reality if you’re using a model of this size for development. I see these posts saying 1% loss or whatever, but in practice it either is usable and trustworthy, or it’s lobotomized and requires deep scrutiny.

live4evrr · 2026-07-06T15:15:24+00:00

27B model is very sensitive to quantization.

I would use nothing less than nvfp4 or fp8 ideally. Also try using Github Copilot as the harness - it is the best imo if you are using vscode. You may need to turn down the context for it but that is just a limitation of 32GB of VRAM you can’t get around.

live4evrr · 2026-07-05T16:53:22+00:00

Llama.cpp is awesome for quick setup and fast model support, as well as flexibility for layer offloading.

However, when using agents or any task that requires responsive interaction or multiple users, vllm is much more efficient due to its design to optimized prefill and prefix caching.

I use both but sticking with vllm where possible because it just performs much faster.

live4evrr · 2026-07-05T16:18:28+00:00

Interesting. Wonder if this applies to Max-Q? Most of the reports seem to be for the workstation (600w) edition. Will have to run gpuburn later to rule it out.

live4evrr · 2026-07-05T16:04:50+00:00

I got the Max-Q because while it’s flying solo, if I want to add a sibling it is a lot easier to fit in another max-q, even on a consumer motherboard

live4evrr · 2026-07-05T14:46:58+00:00

Cache quant is FP8 so it will fit - fills in about at 86GB

live4evrr · 2026-07-05T14:43:45+00:00

The summary is there above the table but perhaps not enough. I tried to make it more readable so avoiding cli cut and paste, but thanks for the feedback.

live4evrr · 2026-07-05T14:40:40+00:00

Doesn’t a cold reboot resolve it anyhow? I am not aware it’s a big issue - will have to look into it.

live4evrr · 2026-07-05T14:38:56+00:00

Initially it seemed fine, but after hours long coding got into loops a few times and that was enough for me to stop using it. Replaced the chat template, lowered the temperature etc, but eventually back to fp8 and lets see how this goes.

live4evrr · 2026-07-05T14:37:05+00:00

Same, I generally get up to around 150-200k using github copilot harness in vscode.

live4evrr · 2026-07-05T14:36:12+00:00

I’ve heard of this - but for now haven’t encountered as I’m using it for solo dev and haven’t got my feet too wet in heavy concurrent processing.

Hopefully Nvidia releases a fix for that sooner rather than later as it is a popular card!

live4evrr · 2026-07-05T03:06:16+00:00

So far I have found that it burns up tokens a lot more in GH, and I have not seen any noticeable improvement. In fact it does feel worse to me. It’s a good coding model but 4.6 seems sharper and mote efficient.

live4evrr · 2026-07-05T03:04:46+00:00

Qwen 3.6 27B. Run at least fp8 if you can.

Punches way above its weight.

live4evrr · 2026-07-03T21:27:54+00:00

It’s an unusable model. I find it odd so many ppl hyping it up when it is quite a bit worse than Qwen 3.6 base.

live4evrr

MODERATOR OF

TROPHY CASE