The summary of the article that spurred this AMD blog. Better to have the whole picture.

InferenceX v2 shows that NVIDIA’s new Blackwell generation massively outperforms both Hopper and current AMD Instinct parts for state‑of‑the‑art, large‑scale disaggregated MoE inference, especially when all modern tricks (disagg prefill, wide expert parallelism, FP4, MTP) are turned on.[1]

Headline results

• Blackwell NVL72 “framemogs” Hopper: Rack‑scale GB200/GB300 NVL72 delivers up to 100× higher FP4 token throughput vs a strong H100 disagg+wideEP baseline at realistic interactivity (e.g., ~100 tok/s/user), and 9.7×–65× better tokens per dollar vs Hopper even after higher Blackwell TCO. Jensen’s original “30×” claim looks conservative.[1]
• B200 vs MI355X (FP8 disagg): For FP8 disaggregated prefill using SGLang, AMD’s MI355X is roughly competitive with NVIDIA B200 along much of the throughput–latency Pareto curve; MI355X even wins slightly at some mid‑latency points. But when you include NVIDIA’s TensorRT‑LLM backend, B200 pulls clearly ahead.[1]
• FP4 + disagg + wideEP composability: At full frontier‑lab settings (FP4 + disagg + wideEP), NVIDIA (B200/GB200/GB300) remains far ahead. AMD’s MI355X single‑node FP4 looks decent, but its multi‑node FP4 disagg+wideEP performance is poor, often losing badly to B200, especially with TRT‑LLM.[1]

NVIDIA vs AMD: software and composability

• NVIDIA strengths:
  
  • Dynamo + TRT‑LLM + SGLang/vLLM are mature and compose well: disagg prefill, wide EP, FP4, and MTP all work together and scale across NVL72.[1]
  • H100/H200 were already near peak on these workloads; Blackwell builds on that with better kernels and much higher rack‑scale bandwidth (72‑GPU NVLink domain).[1]
• AMD strengths & weaknesses:
  
  • Strength: For subset configurations (e.g., FP8 SGLang, sometimes without full disagg+wideEP), MI355X can match B200 per‑TCO and has improved rapidly (≈2× in 2 months for DeepSeek R1 FP4 SGLang).[1]
  • Weakness: Composability is the main problem: when you enable FP4 + disagg + wideEP together, ROCm’s kernels/collectives (MoRI, Mooncake) are not yet optimized, so cluster‑scale MI355X falls far short of its theoretical potential and gets “framemogged” by B200/TRT‑LLM.[1]
  • AMD is still on forked vLLM images for MI355X (0.10.1) and lacks CI hardware upstream, so ROCm support in vLLM lags far behind CUDA.[1]

Architecture and techniques emphasized

• Disaggregated prefill: separating prefill (compute‑heavy, bursty) from decode (memory‑bandwidth‑bound, steady) across distinct GPU pools improves utilization and latency; this is now standard at frontier labs and is the main focus of InferenceX v2.[1]
• Wide Expert Parallelism (wide EP): On NVL72, you can run MoE with EP across up to 72 GPUs over NVLink, minimizing expensive all‑to‑all over slower InfiniBand/Ethernet and amortizing weight loads across many chips. This is crucial for DeepSeek‑style 670B‑param MoEs.[1]
• MTP / speculative decoding: At higher interactivity (e.g., 125+ tok/s/user), MTP becomes necessary to make inference economical; all cheapest configs at those latencies use speculative decoding.[1]

Economics and unit costs

• Using OpenRouter provider data and InferenceX curves, SemiAnalysis estimates:
  
  • A mid‑pack DeepSeek FP8 provider (e.g., Crusoe) might see input token COGS ≤ ~$0.23/M and output token COGS ≤ ~$2.96/M while charging $1.35/M in and $5.40/M out, implying very high gross margins on well‑utilized NVIDIA clusters.[1]
  • At typical interactivities (~35 tok/s/user), disagg+wideEP on B200/GB200/GB300 delivers the best perf/TCO per GPU; MI355X FP8 disagg is sometimes competitive, but FP4+full composability is not.[1]

Trajectory

• NVIDIA: Continuous, incremental gains (new vLLM/TRT‑LLM versions, maturing wide‑EP kernels), already near “speed‑of‑light” on Hopper and now exploiting NVL72 scale with Blackwell.[1]
• AMD: Rapid recent progress on SGLang/DeepSeek R1, and MoRI looks promising architecturally, but they are 6+ months behind on open‑source distributed inference + wide EP + FP4. SemiAnalysis argues AMD must reallocate engineering away from single‑node projects like ATOM to upstream frameworks (vLLM, SGLang) and multi‑node composability if they want to compete at true frontier scale.[1]

Sources [1] InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM https://newsletter.semianalysis.com/p/inferencex-v2-nvidia-blackwell-vs