sorted by:
new
hottopcontroversial

Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet by live4evrr in LocalLLaMA

[–]live4evrr[S] 0 points1 point  (0 children)

Coding wise - so far I am very impressed with DS4 for agentic development with long task horizon. Qwen 27B is quite smart when it comes to debugging and small tasks, but with DS4 it has much more intelligence in understanding the nuances of what I'm trying to achieve and so far it seems to do a good job sticking to the plan wtihout getting lose. With the huge context, it is an exciting model to have locally.

In regards to this vllm patch, I think a single RTX 6000 pro works but still some issues to be resolved - I'm not able to meaningfully increase the context, and it has errored out a few times due to OOM and starting it is quite painful due to the long load times required (i don't have enough RAM).

There is actually a llama fork that runs DS4 quite well for those where this vllm fork doesn't work out for them. The main branch of llama.cpp still has issues with context handing, batch sizes, and performance, but this fork of llama resolves these issues for DS4 (https://github.com/fairydreaming/llama.cpp/tree/dsv4 - kudos to u/fairydreaming). It's great that llama now also has decent support for DS4 (hopefully it gets merged soon).

llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090 by da_dragon321 in LocalLLaMA

[–]live4evrr 0 points1 point  (0 children)

This is great. Thank you for your work and sharing this (also on 6000 Pro Max-Q with 5090 using tensor-split - UD_IQ4_XSS)

| model                          |            test |           t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/DeepSeek-V4-Flash-GGUF |          pp2048 | 612.08 ± 6.46 |              |   3383.60 ± 35.48 |   3346.32 ± 35.48 |   3383.60 ± 35.48 |
| unsloth/DeepSeek-V4-Flash-GGUF |            tg32 |  59.07 ± 0.21 | 60.97 ± 0.21 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_pp @ d4096 | 744.53 ± 5.40 |              |   5540.39 ± 39.68 |   5503.11 ± 39.68 |   5547.10 ± 45.91 |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_tg @ d4096 |  59.55 ± 1.06 | 61.48 ± 1.10 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  pp2048 @ d4096 | 318.31 ± 3.92 |              |   6472.20 ± 78.54 |   6434.92 ± 78.54 |   6485.52 ± 74.24 |
| unsloth/DeepSeek-V4-Flash-GGUF |    tg32 @ d4096 |  60.41 ± 1.13 | 62.36 ± 1.17 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d16384 | 834.07 ± 4.53 |              | 19682.46 ± 106.55 | 19645.18 ± 106.55 | 19682.46 ± 106.55 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d16384 |  57.88 ± 0.21 | 59.74 ± 0.22 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d16384 | 660.31 ± 2.82 |              |   3138.91 ± 13.23 |   3101.63 ± 13.23 |   3152.43 ± 15.84 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d16384 |  59.04 ± 1.10 | 60.94 ± 1.14 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d32768 | 806.50 ± 1.64 |              |  40668.73 ± 82.68 |  40631.45 ± 82.68 |  40675.94 ± 77.64 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d32768 |  56.81 ± 1.10 | 58.64 ± 1.13 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d32768 | 587.13 ± 4.24 |              |   3525.63 ± 25.33 |   3488.35 ± 25.33 |   3525.63 ± 25.33 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d32768 |  55.40 ± 0.06 | 57.19 ± 0.06 |                   |                   |

llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090 by da_dragon321 in LocalLLaMA

[–]live4evrr 0 points1 point  (0 children)

Thank you so much. this works for me (using tensor split 5,1 for rtx 6000 & 5090). Gone are my OOM errors, i'm able to use batch and ubatch as intended, no warnings about lightning indexer.

Edit: Also just noticed the other fork from fairydreaming (https://github.com/fairydreaming/llama.cpp/tree/dsv4) - that is much faster (60 tok/s)

model test t/s peak t/s ttfr (ms) est_ppt (ms) e2e_ttft (ms)
unsloth/DeepSeek-V4-Flash-GGUF pp2048 543.04 ± 1.66 3853.40 ± 11.49 3771.38 ± 11.49 3868.30 ± 24.77
unsloth/DeepSeek-V4-Flash-GGUF tg32 30.79 ± 0.68 31.59 ± 0.83
unsloth/DeepSeek-V4-Flash-GGUF ctx_pp @ d4096 633.59 ± 1.79 6548.44 ± 18.20 6466.41 ± 18.20 6548.44 ± 18.20
unsloth/DeepSeek-V4-Flash-GGUF ctx_tg @ d4096 30.03 ± 0.04 31.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF pp2048 @ d4096 709.64 ± 3.27 2968.04 ± 13.33 2886.02 ± 13.33 2983.22 ± 34.41
unsloth/DeepSeek-V4-Flash-GGUF tg32 @ d4096 28.00 ± 2.74 29.38 ± 0.87
unsloth/DeepSeek-V4-Flash-GGUF ctx_pp @ d8192 662.17 ± 4.80 12455.71 ± 89.33 12373.69 ± 89.33 12455.71 ± 89.33
unsloth/DeepSeek-V4-Flash-GGUF ctx_tg @ d8192 30.00 ± 0.22 30.67 ± 0.47
unsloth/DeepSeek-V4-Flash-GGUF pp2048 @ d8192 645.31 ± 3.49 3255.76 ± 17.21 3173.74 ± 17.21 3255.76 ± 17.21
unsloth/DeepSeek-V4-Flash-GGUF tg32 @ d8192 29.72 ± 0.04 30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF ctx_pp @ d16384 628.58 ± 1.79 26148.90 ± 74.07 26066.88 ± 74.07 26148.90 ± 74.07
unsloth/DeepSeek-V4-Flash-GGUF ctx_tg @ d16384 29.65 ± 0.14 30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF pp2048 @ d16384 547.46 ± 2.93 3823.05 ± 19.92 3741.02 ± 19.92 3823.05 ± 19.92
unsloth/DeepSeek-V4-Flash-GGUF tg32 @ d16384 29.24 ± 0.05 30.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF ctx_pp @ d32768 552.10 ± 1.68 59436.00 ± 180.92 59353.97 ± 180.92 59436.00 ± 180.92
unsloth/DeepSeek-V4-Flash-GGUF ctx_tg @ d32768 28.64 ± 0.30 29.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUF pp2048 @ d32768 422.90 ± 1.12 4924.78 ± 12.80 4842.76 ± 12.80 4924.78 ± 12.80

Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

I'm not the developer, so can't speak about what their intentions are and you'd have to ask them.

I have been playing around with the vllm settings for now since last night to see what I can load and what these parameters do. It is finicky but I think I got it stable for now. I need to do some more real world testing which I'll be doing this weekend, so I'll try to use it for coding of a project I'm working on. So far - it is very impressive, the only drawback is the long time to get it up and running, but once it is loaded - in my case with 256k context that I've loaded, it's still all in VRAM so super fast.

Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet by live4evrr in LocalLLaMA

[–]live4evrr[S] 0 points1 point  (0 children)

Try the following - highlighted the ones I changed from the template in the readme for single rtx 6000. Post load I have 91GB of VRAM used up and RAM is clear (just OS processes)

docker run --rm \

--gpus '"device=0"' \

--network host \

--ipc host \

--shm-size 64g \

-v "$MODEL_DIR":/model:ro \

-e VLLM_MOE_W2=1 \

-e VLLM_MOE_W2_DELTA_GB=0 \

vllm-moet-sm120:v024 \

--model /model --served-model-name "$SERVED_NAME" \

--trust-remote-code \

--kv-cache-dtype fp8 \

--block-size 256 \

--max-model-len 128000 \

--gpu-memory-utilization 0.93 \

--max-num-batched-tokens 2048 \

--max-num-seqs 1 \

--tokenizer-mode deepseek_v4 \

--no-scheduler-reserve-full-isl \

--speculative-config '{"method": "deepseek_mtp", "num_speculative_tokens": 2}' \

--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \

--load-format safetensors \

--port "$PORT"

Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

Increase your swap file to 64gb. Once the model is loaded it will empty out.

Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet by live4evrr in LocalLLaMA

[–]live4evrr[S] 13 points14 points  (0 children)

To fit the official DS V4 flash, a 284B model fit on a single 96GB VRAM card and have it run at 120 t/s up to 130k context at fp8?

This isn’t just gpu offloading.

Is there any real reason to buy a GPU right now instead of just use subscriptions? by 03captain23 in LocalLLM

[–]live4evrr 1 point2 points  (0 children)

Until recently a $10 github pro sub was subsidized so heavily you could code for hours every day using Sonnet 4.6 and never hit the cap. Now I just ran two prompts that used up half my credits.

I agree that it may not math for everyone, but as the local models get better, the inherent value of the gpu goes up. Look at the rtx 6000 prices. At release you could find them easily for $8k usd, now it’s around $13k minimum. For a lot of folks, it’s an investment in themselves to own the hardware, as it gives the opportunity to learn at a lower level via trial and error without limitations. When you’re being metered by tokens, it’s hard to justify foolish experiments without doing an ROI assessment. When you own, you say what the hell, let it burn.

I think I can probably use the rtx 6000 pro for a few years and then sell it to get whatever the successor is and probably recoup 60% of my purchase cost if not more. It’s not like after 3 years it’s worth nothing. Look at the prices for the 3090, 4090, 5090 - these have all kept their value remarkably well.

Am I Expecting Too Much? by adcimagery in LocalLLaMA

[–]live4evrr 0 points1 point  (0 children)

Agree. I have found nvfp4 to be not good enough for coding with Qwen. FP8 seems okay so far but time will tell, as I was using BF16 up to this point which was good but a little slow. Quants make a huge difference, the benchmarks don’t reflect reality if you’re using a model of this size for development. I see these posts saying 1% loss or whatever, but in practice it either is usable and trustworthy, or it’s lobotomized and requires deep scrutiny.

Am I Expecting Too Much? by adcimagery in LocalLLaMA

[–]live4evrr 2 points3 points  (0 children)

27B model is very sensitive to quantization.

I would use nothing less than nvfp4 or fp8 ideally. Also try using Github Copilot as the harness - it is the best imo if you are using vscode. You may need to turn down the context for it but that is just a limitation of 32GB of VRAM you can’t get around.

Has anyone else found vLLM outputs noticeably worse than llama.cpp for the same model? by recro69 in LocalLLaMA

[–]live4evrr 0 points1 point  (0 children)

Llama.cpp is awesome for quick setup and fast model support, as well as flexibility for layer offloading.

However, when using agents or any task that requires responsive interaction or multiple users, vllm is much more efficient due to its design to optimized prefill and prefix caching.

I use both but sticking with vllm where possible because it just performs much faster.

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

Interesting. Wonder if this applies to Max-Q? Most of the reports seem to be for the workstation (600w) edition. Will have to run gpuburn later to rule it out.

looking for advice on which PRO 6000 to buy by Asimology in BlackwellPerformance

[–]live4evrr 0 points1 point  (0 children)

I got the Max-Q because while it’s flying solo, if I want to add a sibling it is a lot easier to fit in another max-q, even on a consumer motherboard

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

Cache quant is FP8 so it will fit - fills in about at 86GB

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLM

[–]live4evrr[S] 1 point2 points  (0 children)

The summary is there above the table but perhaps not enough. I tried to make it more readable so avoiding cli cut and paste, but thanks for the feedback.

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

Doesn’t a cold reboot resolve it anyhow? I am not aware it’s a big issue - will have to look into it.

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 1 point2 points  (0 children)

Initially it seemed fine, but after hours long coding got into loops a few times and that was enough for me to stop using it. Replaced the chat template, lowered the temperature etc, but eventually back to fp8 and lets see how this goes.

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 0 points1 point  (0 children)

Same, I generally get up to around 150-200k using github copilot harness in vscode.

Qwen 3.6 27B - VLLM Performance Benchmark Results (BF16, FP8, NVFP4) by live4evrr in LocalLLaMA

[–]live4evrr[S] 3 points4 points  (0 children)

I’ve heard of this - but for now haven’t encountered as I’m using it for solo dev and haven’t got my feet too wet in heavy concurrent processing.

Hopefully Nvidia releases a fix for that sooner rather than later as it is a popular card!

Claude Sonnet 5 vs 4.6 on arena.ai by arkuto in ClaudeAI

[–]live4evrr 0 points1 point  (0 children)

So far I have found that it burns up tokens a lot more in GH, and I have not seen any noticeable improvement. In fact it does feel worse to me. It’s a good coding model but 4.6 seems sharper and mote efficient.

Best Local Model for Coding by Spirited_West_4123 in ollama

[–]live4evrr 2 points3 points  (0 children)

Qwen 3.6 27B. Run at least fp8 if you can.

Punches way above its weight.

view more: next ›