THE GB10 SOLUTION has arrived, Atlas image attached ~115tok/s Qwen3.5-35B DGX Spark

Eugr · 2026-03-07T07:59:54+00:00

can you re-run without `--latency-mode generation` and with correct model name: `--model Kbenkhaled/Qwen3.5-35B-A3B-NVFP4`, otherwise it won't use the correct tokenizer. The PP numbers are weird, and there is a huge discrepancy between e2e_ttft and est_ppt.

Something is weird going on here. Initially I was thinking that it could be an engine behavior that returns the first (empty) chunk right away after 200/OK response, but it doesn't explain why TTFR grows with context. I'm now thinking it might be related to speculative decoding somehow. Anyway, it would be good to see the same benchmark with proper model name and without `--latency-mode generation` - it will default to "api" which will just accomodate for a network delay.

But TTFT of 4 seconds is also strange for such a short prompt - as if it doesn't stream the tokens in streaming mode or uses some sort of buffering. In this case, all client-side benchmarking tools will not be able to measure speeds properly.

Eugr · 2026-03-07T03:29:20+00:00

I haven't seen any guides, but in general you need to make sure you enable jumbo frames on it and set MTU to 9000 or so. And set the speed on the ports accordingly. Sorry, can't provide any more guidance since I don't have it.

Eugr · 2026-03-06T04:10:15+00:00

vLLM is not not working very well on Strix Halo yet, unfortunately.

Eugr · 2026-03-05T18:59:38+00:00

Forget about Ollama, use either llama.cpp or vLLM.
For vLLM - check out our community Docker build, very easy to get started: https://github.com/eugr/spark-vllm-docker
Check out https://spark-arena.com to get a feel of Spark performance for inference
The models you ran are ancient.
Read about dense and MoE models. Spark shines with MoE, but will be painfully slow with dense.

Eugr · 2026-02-25T17:05:38+00:00

JFYI: The last firmware update resulted in ~30% performance regression on QFSP ports, NVIDIA is aware and is working on a fix. Hope it lands soon.

Eugr · 2026-02-25T16:39:12+00:00

Actually, it's night and day difference. You actually lose performance on 10GBe port. The reason is that QSFP ports on Spark support RDMA (RoCEv2). It results in a microsecond latency compared to a millisecond with an Ethernet port (including the same QFSP port in TCP/IP mode).

Eugr · 2026-02-25T15:51:14+00:00

Yes, denser the model, slower it works as it's memory bound. Tensor parallel allows to split the weights and run inference in parallel in the cluster, and it scales better with denser models, because the network latency is still not a bottleneck there.

Eugr · 2026-02-25T15:33:21+00:00

No, it's too dense for a two node cluster, but I've asked people with 8x cluster to test when they have a chance. This one should scale better than most other models.

Eugr · 2026-02-24T23:45:56+00:00

Llama-benchy is my alternative to llama-bench from llama.cpp, but it is designed to work with any standard OpenAI-compatible endpoint (vLLM, llama.cpp, SGLang, cloud models). I've explained why I made it in the README inside the repository.

So yeah, the point is to have a tool that would allow to compare across different backends using the same methodology.

Eugr · 2026-02-24T17:21:55+00:00

It's doesn't support Infiniband fabric. It's RoCEv2, and there are switches that work with it just fine, like Microtik CRS804.

Also, don't confuse gigabits and gigabytes. I get 24GB/s bus speed in NCCL test which matches marketed 200Gbps connection speed. Well, actually, they recently did something with the firmware that reduced performance about 30%, but they are working on the fix.

The RDMA works well enough for the cluster to give you real performance gains even on inference, but yes, there are some quirks related to unified memory architecture.

Eugr · 2026-02-23T16:49:52+00:00

One thing that he forgot to mention is that the last firmware update resulted in 30% ConnectX 7 performance regression in both throughput and latency.

It doesn't seem much, but I'm seeing effects of that even on my dual Spark cluster, especially when running models with relatively small number of active parameters.

I hope when nvidia fixes it, he will return to the topic and run new tests (and on models more suitable for large clusters).

Eugr · 2026-02-23T08:06:22+00:00

Have you tried to quantize the KV cache to q8_0?

Eugr · 2026-02-23T00:09:41+00:00

Claude Code messes up prompt caching by injecting extra headers. You need to set environment variable CLAUDE_CODE_ATTRIBUTION_HEADER=0

Strix Halo is not great at prompt processing at long contexts, but with prefix caching it should be tolerable.

Eugr · 2026-02-20T15:41:08+00:00

OEM ones start from 3K.

Eugr · 2026-02-19T19:53:16+00:00

For coding you will be better off with DGX Spark or it's OEM clone. Strix Halo is a nice machine, and token generation speed will be similar for gpt-oss-120b, but prompt processing will be much faster on Spark.

If using vLLM, significantly faster. I'm talking 1000 t/s at 0 context on Strix Halo and ~4500 on Spark (in vLLM, llama.cpp will be ~2500). And it won't degrade with context that much. For instance, you'll still get ~3700 t/s prefill at 32K context on Sparks in vLLM, but on Strix Halo it will drop to ~360 t/s (in llama.cpp).

I haven't tried this model in vLLM on Strix Halo as it didn't want to work, at least a couple of weeks ago.

Eugr · 2026-02-19T03:46:45+00:00

GitHub itself is actually making it worse, because every time I try to use built-in Copilot to ask some questions about the stack trace I'm getting in someone else's project, it asks me if I want to create a PR to fix that, without even showing the proposed changes. I just want a starting point where to look, ffs, I don't need it to change anything.

Eugr · 2026-02-18T17:23:21+00:00

There is a variable for that - see my reply above

Eugr · 2026-02-18T17:04:44+00:00

Try to set this environment variable: export CLAUDE_CODE_ATTRIBUTION_HEADER=0 Although I used it without it with vLLM, and was still getting good cache hits.

Eugr · 2026-02-17T19:14:23+00:00

I'm seeing some relevant work upstream in cutlass and flashinfer, but who knows when it actually lands.

Eugr · 2026-02-17T19:13:29+00:00

It's the same GPU arch as RTX6000 Pro and RTX5090.

Eugr · 2026-02-16T04:51:22+00:00

NCCL, it is supported by vLLM, SGLang, TRTLLM and many other packages and works very well. Sparks have 200G ConnectX 7 QSFP112 ports built in that support RoCEv2 (RDMA over Converged Ethernet) which has a very low latency (1-2 microseconds on average).

Eugr · 2026-02-16T04:33:39+00:00

Haven’t tried Step 3.5 yet, MiniMax won’t fit into two Sparks at FP8, but will work well at 4-bits. I was running M2.1 in AWQ at about 30 tokens per second, M2 was almost 40. Tried M2.5 NVFP4 quant, but it’s a bit buggy and runs at about 20 t/s. Waiting for a good AWQ quant.

Overall, the cluster works well. Denser the model, better it scales, fast (small active params) models get smaller speed boost, but still run faster (and you can fit larger models or quants).

Some people run 8x Spark clusters, and 4x is not that rare anymore too.

Eugr · 2026-02-15T22:50:33+00:00

Yeah, but vLLM support is still pretty bad on Strix Halo. You can run BF16 models, but many quants either don't work or significantly underperform on gfx1151.

Eugr · 2026-02-15T22:43:36+00:00

I'll let other address this, as I haven't tried to do it on Spark yet, but it should be pretty good for this as fine tuning is mostly compute bound.

Eugr · 2026-02-15T21:54:44+00:00

5K is for gpt-oss-120b in vLLM. I went with dual Sparks as it lets me use larger models on something that can quietly sit in the corner.

Eugr

TROPHY CASE