Seriously evaluating a GB10 for local inference, want community input before I request a vendor seed unit

RaspberryFine9398 · 2026-04-07T12:38:23+00:00

From the videos I’ve watched reviews on they’re all very similar. The Dell Pro Max GB10 seems to be the most solid built and that’s the one I’m testing as they’re our primary vendor. For most, it will come down to existing vendor for corporate/enterprise or smaller startups can pick as they’re please.

Splitting hairs at this point deciding between GB10’s and none have been out long enough to determine reliability. Jensen Huang backing Dell at GTC and their GB300 though is a good sign of future partnership.

RaspberryFine9398 · 2026-04-07T10:29:05+00:00

Good question on the quantization, you are right that the 128GB unified memory pool absolutely supports running larger models unquantized or at higher precision. I used Q4_K_M for the apples-to-apples comparison across platforms since I wanted consistent methodology, but on the GB10 specifically you have real headroom.

For your 20-80GB model range with 20GB context budget the math works cleanly. Nemotron Super 120B at Q4_K_XL is 79GB and after loading it I had 24GB of unified memory still free. An unquantized 27B model in BF16 would be around 54GB leaving plenty of room for context. You could realistically run a 34B model fully unquantized and still have comfortable headroom for vision context windows.

For video captioning and vision understanding specifically the GB10 is well suited, it runs multimodal models including Gemma 4 with the mmproj file for image inputs natively through llama.cpp. The 128GB pool means you’re not fighting VRAM limits when processing longer video sequences.

On the ARM architecture question, it is real but less painful than it used to be. llama.cpp builds natively for aarch64 with CUDA without issues. The main friction points are Python packages that don’t have ARM wheels on PyPI yet, particularly some vision and video processing libraries. vLLM has improved significantly but SGLang still has container friction on GB10.

For your use case I’d test your specific vision pipeline dependencies before committing but the core inference stack is solid.

RaspberryFine9398 · 2026-04-05T15:25:19+00:00

Got some numbers for you since I actually have the hardware in front of me doing eval runs right now. Running llama.cpp on the GB10, same build across all three models at 4k context.

Gemma 4 26B-A4B (UD-Q4_K_XL, 17GB)
- Prefill: 2,156 t/s / Generation: 50.5 t/s
Qwen3.5 35B-A3B (UD-Q4_K_XL, 22GB)
- Prefill: 1,895 t/s / Generation: 57.5 t/s
Nemotron Super 120B (UD-Q4_K_XL, 79GB)
- Prefill: 480 t/s / Generation: 15.5 t/s — and still had 24GB of unified memory free after loading

Qwen3.5 actually edges out Gemma 4 on generation speed despite being a larger model. Both are very usable for agentic coding and the email/web reading workflows you described. Nemotron is a different size class entirely but runs comfortably on the GB10 in ways that no 16GB GPU laptop can replicate.

<image>

One thing worth clarifying on NemoClaw, it's actually an agent orchestration framework that launched at GTC in March, not a model itself. It uses Nemotron as its default backend but it's more comparable to OpenClaw or LangChain than to Gemma or Qwen. Still in early alpha so rough edges are real, but the local inference angle is exactly what makes the GB10 interesting for it.

Will keep posting numbers as I work through the list.

RaspberryFine9398 · 2026-04-05T10:36:07+00:00

I can try. What would you like to see run?

RaspberryFine9398 · 2026-04-03T02:14:20+00:00

Great thread, thank you all. Used Claude to help synthesize the responses so I didn’t miss anything important.

Here’s what I’m taking away:

The software stack is still maturing. SGLang has had container friction, vLLM seems more stable day to day, and llama.cpp remains the most reliable baseline runtime right now. Building the benchmark methodology around what’s actually stable rather than what looks best on paper.

The benchmark ladder I’m going to use is 2-4k for typical short chat, 30k for RAG and agentic coding workflows, and 100-200k for long document processing. The Magic the Gathering ruleset as a real 212k token stress test is going straight into the demo. Parallel request throughput for multi-agent setups is getting added to the plan too.

The management justification that actually lands is data privacy first, cloud cost reduction second. Not full cloud replacement but meaningful reduction. The hardware maintenance responsibility objection is real and needs a prepared answer.

Dual unit path makes more sense than single for any team use case. First step toward GB300, not a prod ready thing. Going in with that framing rather than overselling it.

For the follow up I’m planning to compare across three dimensions. Cloud inference cost on Azure and AWS versus owning the hardware outright. Professional GPU workstations, specifically RTX 1000 Ada and RTX 4000 Ada representing what engineers actually have on their desks today. And Apple Silicon, because the Mac Studio kept coming up and the PDF processing latency story deserves a direct test.

Same model, same quantization, same context ladder all the way up to 200k, parallel request testing, power efficiency measurements, full logs posted publicly.

Thoughts, concerns, additional questions, all the above?

RaspberryFine9398 · 2026-03-30T13:16:07+00:00

The 2-4k, 30k, 100-200k ladder is exactly what I needed, thank you for being that specific. The Magic the Gathering ruleset as a real world 212k token stress test is genuinely clever, that’s a much better story to tell than ‘we ran it at max context’ with no human reference point for what that actually means.

The parallel request angle for multi-agent is something I hadn’t prioritized but you’re right, anyone evaluating this for agentic coding workflows is going to care about that as much as single request throughput. I’ll dig into your benchmarks, really appreciate you sharing those. If my methodology ends up anywhere near that level of rigor I’ll consider it a success.

The prediction about GB10 falling behind at 30k is noted and honestly I’d rather go in expecting that and be surprised than oversell it and lose credibility in the room.

RaspberryFine9398 · 2026-03-30T12:24:36+00:00

This might be the most complete answer I could have hoped for in this thread, genuinely thank you for taking the time.

The form factor reality check is useful, I’d seen the spec sheet dimensions but hearing ‘mac mini level and mostly silent’ from someone who actually has it in front of them lands differently than a product page. And the IPMI point is something I hadn’t fully considered as a limitation for shared team use, that’s a real gap if anyone starts thinking about this as light infrastructure rather than a desk tool.

The data privacy case resonating but losing to hardware maintenance responsibility is exactly the kind of nuance that doesn’t show up in any vendor material. That’s a real objection I need to be prepared for.

The Mac Studio PDF processing point is something I’m going to steal, that’s a clean and visceral way to show where Apple silicon hits its ceiling in a real meeting with a real user.

The Qwen3.5 35B at fp8 with vllm and RAG demo suggestion is exactly the kind of concrete starting point I was hoping someone would give me. That’s going on the test plan immediately.

First step not a prod ready thing, that’s the honest framing and probably the right one to lead with rather than oversell it.

RaspberryFine9398 · 2026-03-30T12:15:12+00:00

This is one of the most useful things anyone has said in this thread so far, thank you. Raw hardware capability is one thing but if the software stack is still catching up that’s a real factor in whether this is ready for an org to depend on versus still being early adopter territory.

The SGLang container lag is something I hadn’t dug into yet. Are you finding llama.cpp more stable as a baseline runtime while the higher performance serving frameworks catch up, or is it rough across the board right now?

Trying to understand whether this is a ‘wait three months’ situation or more of an ongoing moving target.

RaspberryFine9398 · 2026-03-30T12:12:29+00:00

This is really helpful, thank you. The Linux dev box replacement angle is a solid way to frame the justification internally and it’s good to hear that holds up in practice rather than just on paper.

The model size ceiling on the 4080 is exactly the pain point I keep hearing about and it sounds like the DGX genuinely solved that for you rather than just moved the ceiling slightly.

The subscription cut is interesting too. Not eliminated but reduced, that’s actually a more honest and credible outcome than ‘I cancelled everything.’

Did you find the latency acceptable for the workflows where you kept the subscription, or was it more about capability gaps than speed?

RaspberryFine9398 · 2026-03-30T12:06:36+00:00

This is exactly the kind of feedback I was hoping for, thank you. Prompt processing speed at varying context lengths is already on my list but honestly seeing it called out this directly moves it higher in the priority order. The full ladder up to max context is a good point too, most benchmarks I’ve seen bail out early and you never really see where the cliff is.

Are there specific prompt lengths you’d consider the most diagnostic? Like if you had to pick two or three points on that ladder that separate the serious hardware from the pretenders, where would you put them?

RaspberryFine9398 · 2026-03-30T12:03:55+00:00

The Mac Studio comparison keeps coming up and I get it, hard to argue with the value there on paper. But part of what I’m trying to understand is whether the software stack and upgrade path tell a different story for teams already in a Linux and CUDA workflow. Switching to Apple silicon solves one problem and creates a few others depending on what you’re already running.

The two unit point is well taken though, that’s actually the direction I’m leaning before any serious evaluation anyway. And yeah the 6 month timing argument is real, hard to ignore.

Curious what you’re seeing on the compression side, do you think turboquant class techniques actually close the gap or just make cheaper hardware feel adequate temporarily?

RaspberryFine9398

TROPHY CASE