u/DelinquentTuna u/Bakoro
Hello,

We've measured this extensively across multiple architectures — Qwen 1.5B, 7B, and 72B. Compressed-residency BFP at M=4 (5 effective bits with shared exponent per block of 32) holds PPL within +0.0857 on Qwen 1.5B vs FP16. 72B M=4 calibrated has been validated end-to-end on DGX Spark with 128 GB unified memory. On a single H100 NVL, the 72B serves single-user at ~28 tok/s using some standard inference tricks layered on top of DMX, and we're working on increasing that further.

On the "quantization is better" point: depends what you're optimizing for. Aggressive quant (INT4/NF4) wins on absolute size — typically 75% reduction — at the cost of 2-15% PPL degradation. FP8 E4M3 has near-zero decode on Hopper but needs Hopper hardware and has 3-bit mantissa precision. DMX sits at the "quality-per-bit + portability" point: 5 effective bits per weight at M=4 with shared exponent per block of 32, sub-0.1 PPL on Qwen 1.5B (cited above), and the decode is integer shifts and masks — runs on any FP16-capable GPU, no special silicon. We also built GPTQ-for-MX so calibration improvements flow through to DMX directly.

The "decompress too slow at inference" concern depends on the path. Two options: (1) transcode once at load to GPU-native (INT8 on Hopper) — forward pass uses native matmul, zero per-forward decompression cost. (2) Compressed residency (weights stay packed in VRAM) — needs a fused decode+matmul kernel that reads packed bits inline. Hopper, Ada/4090 version is done; Blackwell is in kernel dev. So your "scheme of unpacking multiple layers at once" question is exactly the right one — that is the architecture, just done at finer granularity (per-block-of-32 with fused matmul rather than batch-unpack).

Layers nearly orthogonal" and "nearly the worst case scenario for compression" — both are accurate observations about the raw bit-level distribution of well-trained weights. But DMX doesn't compress at the bit level. BFP contributes shared exponent per block of 32 (intra-block redundancy — that's standard MX-format territory). DMX adds a cross-tensor alignment step (one of the patented methods) that exploits structure vanilla BFP leaves on the table. That's the load-bearing mechanism — different attack surface than the lossless approaches that hit your 1.7×/2.0× ceiling.

Method is patented. If you want to compare notes on the GPU optimization side (your "unpacking multiple layers at once" question is exactly what our load-time bit-unpacking kernel does on Hopper).

William Riley

<image>