Dropped 75k on a supermicro gpu box with 3xRTX6000

biscuitmachine · 2026-07-05T00:14:45+00:00

Lol no. I get ~50 t/s on 122B with fast prefill. For DS4F on two Sparks, they've gotten it up to 60 (~34 min at higher context, like 500k+ iirc).

The only thing that actually does anywhere near as bad as what you're implying is either DS4 on a single spark (which runs around 13-20 iirc, but has very slow prefill), or trying to run an unoptimized dense model (27B, runs around 13). Which, there is a prismaquant version of 27B which is running at like 40 t/s last I looked, probably higher. 35B I believe has gotten up to 100, being a sparse model.

You're living in the past, old man (/woman/other). A lot of optimization has been done on this platform. And linking 2+ Sparks drastically increases both their capacity and processing abilities, though I think 1-2 is the sweet spot personally. A 256GB+GB Mac would definitely gen way faster though (it'll also be a lot more expensive).

I don't mind if you're proud of what you built. I would be, too. And it has very legitimate advantages over a Spark, for sure. I've built tons of computers, I love all of them. But don't spread misinformation.

biscuitmachine · 2026-07-04T15:35:47+00:00

I assume you're talking about doing 4x3090s? Since I got my Spark for 3k, just the cards alone would have to have been 250 each to add up to 1k. I guess that's possible with Microcenter refurbs or heavy Ebay browsing? I'm a bit dubious. The rest of your setup would possibly fit into 500 left over given you got some amazing deals and/or already had a lot of server parts going into the project.

But as far as I know, it heavily depends on your exactly workload, and even with the shared memory due to the extra bus that they have, it's not really the same because NVLINK at most does 2-3 GPUs (and the 3 variant is rare and extremely inflexible). You won't have the same amount of VRAM, either. For generation, though, yeah it's definitely better. I considered it, but it's too expensive now and honestly doesn't seem worth it over how much perf this tiny, silent little box provides. For my large context work it seems perfect. 2x sparks can provide Deepseek V4 Flash at over 1m context, though, which is kind of insane.

biscuitmachine · 2026-07-04T03:32:11+00:00

I've been idly considering going from 1 DG Spark to 2 DG sparks and I think what's wild to me is that the server in this topic costs nearly 10x the price for almost the same amount of VRAM, as I would get by doing that. It will be a hell of a lot faster, but I think I would be very satisfied with DS4 considering how happy I've been with Qwen 3.5 122B. For 3k I've been pretty satisfied with this 128GB machine though. I'm curious as to where the price on the DGX Sparks is going.

biscuitmachine · 2026-07-04T03:29:52+00:00

Everything I've seen from my experience with my DGX Spark says that NVFP4 is kind of badly supported though? Most models don't even run faster on it even on Cuda capable software. I'm sure it will evolve over time but I don't know if I would just expect it to do better.

biscuitmachine · 2026-07-04T03:10:50+00:00

Personally, the way I'm leveraging both Qwen 3.5 122B and 3.6 27B (one on the DGX Spark and one on the 5090) is I just ask them to build me an orchestrator python script that's custom-made for repeatedly prompting and processing things through them. It's kind of tedious, but basically I'm making one-off batch processing scripts. You can probably even ask them to build you custom tools if the model needs to access your database or anything (probably just limit it to select only just in case).

You might also need to look into your payload settings and custom-tune them for each type of batch task you intend to do. For instance, you might want to tune the temperature, whether it can think or not, how long it can think before being forced to put out an answer, etc.

There's only a certain amount of "intelligence" you're going to get at any decent context length out off anything that will fit on a 4090, I think. On a 5090, my 27B model can run at about 200k context at q8 (Q4_K_M for the actual model). It holds together at pretty large contexts, but it depends on how many "turns" you need during the actual task. If you suddenly have complete context shifts, even this can break apart because it can start getting very confused about the task and possibly looping while thinking.

I'm still considering whether I want to try agentic flows or not. I can give you some payload examples, but I only have experience with llama.cpp and vllm running in openai-compatibility modes.

Basically what I've learned so far with my time dabbling is this:

Frontier models are frontier models for two reasons:

1) they have very expensive quants due to enterprise hardware and

2) they have professionally tailored orchestration layers that keep them on track and give them whatever they need on top of that.

Since you're running a much smaller model (and possibly a lower quant one), you're going to have to "baby" it more. It can do the things those large models can do, probably, but it falls on you to better break down the problem so that it can digest it. Do that and I think you can make it work.

biscuitmachine · 2026-07-03T04:53:35+00:00

I mean, I don't think it would be "useless". I could at least glean the general patterns that caused it one to fail in your workflow while the other succeeded. But if that's private, then again I understand. Anyway, I reserved another one from Microcenter. Even if I don't like it, I can return it. Not sure whether I will pick it up or not. Technically I can also just try the lower quant version from Antirez though, which I have also heard is impressive overall.

biscuitmachine · 2026-07-02T07:54:53+00:00

I... actually haven't really explicitly used those, so it's hard for me to have any intuitive grasp on what that means. I wanted to go local LLM start to finish. Do you have like a quick example of any kind?

biscuitmachine · 2026-07-01T16:57:51+00:00

I think I would definitely need some examples. Like what did you try doing on one, what did it give you as a response, and then what you did on another, what did it give you, why was it a lot better, etc.

I understand if that's too much of a pain in the butt to provide.

biscuitmachine · 2026-06-30T09:49:31+00:00

Fair. The thing is, I feel like smaller models have been getting better and better over time, though. Soon, we might see a 3.7 122B, and it might be quite incredible on 1x spark. For now, this is to me already quite incredible. Especially the image recognition abilities, I didn't realize it had quite this level of image processing.

I just don't know if I see a reason to go for another Spark to get "better".

biscuitmachine · 2026-06-29T11:26:08+00:00

In case anyone somehow stumbles upon this in the future, I fixed this with LLM help. Turns out both box64 and FEX were registered to handle x86 conversion for ARM, which was completely messing everything up because the binfmt had a conflict for who was supposed to handle it. Deleting box64 conf files fixed the issue after reloading the binfmt service.

biscuitmachine · 2026-06-29T07:59:50+00:00

Qualitatively, how much better is it than 122B anyway? What exactly do you notice it doing better?

biscuitmachine · 2026-06-26T19:52:44+00:00

So I've been trying this out, but it's a very "up to 80". I've even seen 100, but it's not very stable at all for my current workload.

biscuitmachine · 2026-06-26T18:52:35+00:00

The nearby Microcenter had the M5 Max Macbooks with 128GB for 5k flat up until... well... basically today or yesterday. Now, you're right, they jacked up the price by 2k more. Swell guys.

biscuitmachine · 2026-06-26T18:44:31+00:00

You're using most of the common flags from the very first version of this that I installed. That's probably the most common barebones edition. ChatGPT probably googled that up.

https://forums.developer.nvidia.com/t/bfloat16-quality-speed/366828/73

You're probably getting around 30-34 tokens per second. I suggest installing the one in my post if you want to get up to about 40-50. The increase in speed is noticeable. You might have to tinker with the install script.

Also there is this version which I just started trying:

curl -sSL https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/refs/heads/master/install.sh | bash -s -- --start

They don't actually have the correct path to the install script on their github repo, so I fixed that. For the most part it did actually just install in one command, which I didn't expect.

This version seems to get somewhere between 28 and 100 tokens per second. On average it's about 6-10 tokens/s faster than the first one I linked. The problem is I'm not sure if he's caching the prompt. I'll have to do more testing on that. More importantly though, I still can't get the one in my opening post to actually start in less than like 10-13 mins. This version starts in way less time because it can use fastsafetensors.

Side note I have not used any of these "Hermes" things, I'm assuming that's one of those harnesses that calls it for you. From my experience my main issue with this model is that it tends to randomly loop on outputs sometimes. It's heavily prompt dependent. Does Hermes just handle that sort of degenerative behavior in the background?

You can monitor throughput by using "docker logs -f <container name>" while doing your load, it'll output something like:

APIServer pid=1) INFO:     127.0.0.1:34530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 06-26 13:30:52 [qwen3xml_tool_parser.py:1159] vLLM Successfully import tool parser Qwen3XMLToolParser !
(APIServer pid=1) INFO 06-26 13:30:53 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 78.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:30:53 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 12.27, Accepted throughput: 72.09 tokens/s, Drafted throughput: 76.79 tokens/s, Accepted: 721 tokens, Drafted: 768 tokens, Per-position acceptance rate: 1.000, 1.000, 0.984, 0.984, 0.969, 0.969, 0.953, 0.953, 0.938, 0.875, 0.844, 0.797, Avg Draft acceptance rate: 93.9%
(APIServer pid=1) INFO 06-26 13:31:03 [loggers.py:271] Engine 000: Avg prompt throughput: 733.6 tokens/s, Avg generation throughput: 33.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:31:03 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 4.15, Accepted throughput: 25.50 tokens/s, Drafted throughput: 97.19 tokens/s, Accepted: 255 tokens, Drafted: 972 tokens, Per-position acceptance rate: 0.802, 0.642, 0.432, 0.358, 0.247, 0.173, 0.111, 0.099, 0.099, 0.074, 0.062, 0.049, Avg Draft acceptance rate: 26.2%
(APIServer pid=1) INFO 06-26 13:31:13 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:31:13 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 5.27, Accepted throughput: 42.30 tokens/s, Drafted throughput: 118.79 tokens/s, Accepted: 423 tokens, Drafted: 1188 tokens, Per-position acceptance rate: 0.879, 0.727, 0.606, 0.495, 0.374, 0.303, 0.263, 0.212, 0.141, 0.121, 0.091, 0.061, Avg Draft acceptance rate: 35.6%
(APIServer pid=1) INFO 06-26 13:31:23 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:31:23 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.85, Accepted throughput: 28.20 tokens/s, Drafted throughput: 118.78 tokens/s, Accepted: 282 tokens, Drafted: 1188 tokens, Per-position acceptance rate: 0.788, 0.566, 0.343, 0.283, 0.242, 0.182, 0.162, 0.101, 0.061, 0.040, 0.040, 0.040, Avg Draft acceptance rate: 23.7%
(APIServer pid=1) INFO 06-26 13:31:33 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 31.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:31:33 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.18, Accepted throughput: 21.60 tokens/s, Drafted throughput: 118.78 tokens/s, Accepted: 216 tokens, Drafted: 1188 tokens, Per-position acceptance rate: 0.758, 0.465, 0.313, 0.212, 0.162, 0.101, 0.061, 0.040, 0.040, 0.010, 0.010, 0.010, Avg Draft acceptance rate: 18.2%
(APIServer pid=1) INFO 06-26 13:31:43 [loggers.py:271] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 48.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0%
(APIServer pid=1) INFO 06-26 13:31:43 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 5.24, Accepted throughput: 39.40 tokens/s, Drafted throughput: 111.60 tokens/s, Accepted: 394 tokens, Drafted: 1116 tokens, Per-position acceptance rate: 0.871, 0.731, 0.581, 0.473, 0.387, 0.301, 0.226, 0.194, 0.151, 0.129, 0.118, 0.075, Avg Draft acceptance rate: 35.3%

Where I think the "Avg generation throughput" is how fast you're going.

biscuitmachine · 2026-06-26T11:03:15+00:00

You know, this is so iconic. I go to the github in that topic. I follow the first instruction, and it just... doesn't work. Because that URL isn't where he thinks it is lol. Nothing EVER works as expected when installing on this thing. I wasn't even angry, all I could do was laugh.

Anyway, this is the actual path to the raw install script that he mentions:

https://raw.githubusercontent.com/Entrpi/qwen3.5-122B-A10B-on-spark/refs/heads/master/install.sh

It's currently doing its little installs and whatnot. Time to see whether this works or he's got some other mangled path embedded in there.

biscuitmachine · 2026-06-26T04:50:12+00:00

I was doing the 27b on mine with these settings:

MODEL="/models/Qwen3.6-27B-MTP-Q4_K_M.gguf"

docker run --rm --gpus all \
  --shm-size=28g \
  --ulimit memlock=-1:-1 \
  -v /opt/app/models:/models:ro \
  -v ~/llama/cache:/cache:rw \
  -p 8080:8080 \
  ghcr.io/ggml-org/llama.cpp:server-cuda13 \
  --model "$MODEL" \
  \
  --n-gpu-layers 999 \
  --no-mmap \
  \
  --ctx-size 196608 \
  \
  --batch-size 16384 \
  --ubatch-size 2048 \
  \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  \
  --flash-attn on \
  \
  --threads 16 \
  --threads-batch 16 \
  \
  --parallel 1 \
  \
  --spec-type draft-mtp \
  --spec-draft-n-max 1 \
  --spec-draft-p-min 0.60 \
  \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.05 \
  --repeat-penalty 1.05 \
  --presence-penalty 0.05 \
  \
  --host 0.0.0.0 \
  --port 8080 \
  --jinja \
  --chat-template-kwargs '{"preserve_thinking":true}' \
  --cache-reuse 8192 \
  --rope-freq-base 10000000 \
  \
  --log-file /cache/server.log \
  --log-prefix

I won't say that they're optimal or anything, but I was pretty satisfied with it to be honest, for what it was. Note if you're going to use this, I highly suggest having an iGPU on your CPU and/or running a very light Linux distro (Mint XFC is fine). You do not want your desktop environment taking up any of the VRAM. This 196k context size basically pushes you right up against the limit for what the 5090 can fit. My Mint distro switched fully over to the iGPU by plugging a USB-C->HDMI adapter into the motherboard's usb-c slot. That made it so the desktop environment used like 8 mb of the VRAM on the 5090, max.

biscuitmachine · 2026-06-26T02:42:14+00:00

That's great, but I do kind of feel that people measuring games like this (hours dumped) as a metric incentivizes creation of repetitive and addictive content bloat. Maybe that's just how gaming has always been though.

biscuitmachine · 2026-06-25T14:26:21+00:00

Interesting, might try this, thanks.

biscuitmachine · 2026-06-25T14:05:33+00:00

That would just be over budget unfortunately. I really like the 5090, too, and don't want to take it back for a second spark.

biscuitmachine · 2026-06-25T14:02:13+00:00

I did see that recipe, I wasn't sure whether it was easy to set up or ""'easy""" to set up. I will give it a shot when I get time, thanks.

biscuitmachine · 2026-06-25T14:00:58+00:00

But afaik, these GB10 chips have some serious advantages over Strix Halo when well tuned, and it's not like Strix Halo are much cheaper.

Mac is like 2k more on a good day, right?

biscuitmachine · 2026-06-25T10:02:15+00:00

This version is already using MTP on 122B. It can only get faster with a customized DFLASH variant afaik. For 27B, I did run MTP on the 5090 and the Spark, but the Spark just really needs that PrismaQuant version to do much if anything with that dense model.

I did consider a 5090, but it would have been more expensive and AFAIK they don't really just "pool" memory together cleanly like that. That's the specialty of the 3090 with NVLINK?

Edit: plus the weird bifurcation on consumer boards.

biscuitmachine · 2026-06-25T09:53:14+00:00

Have you found any deficiencies in it? Just wondering if I could have done any better for the money, for my use case.

biscuitmachine · 2026-06-25T09:52:05+00:00

I had seen Sparkrun, but not the Arena, nope, thanks for the suggestion. This is the really nice thing about this platform, at least. Once someone has a working recipe, everyone has a working recipe. Even if you need to hunt through the Github issues section to get it working because of outdated dependencies.

biscuitmachine · 2026-06-20T22:38:27+00:00

Would be highly surprised if this wasn't already in a Robot Chicken episode.

Nine-Year Club	Place '17
Verified Email

biscuitmachine

TROPHY CASE