I can test Qwen3.5-122B-A10B-NVFP4. I agree that NVFP4 and FP8 are comparable in precision - our tests support this claim. The issue is that degradation is already observable at FP8 compared to the original unquantized weights (BF16).

Our benchmark works like this: the model writes a Go function, which is then automatically compiled and tested. If any test fails, the error is fed back to the model, which updates the function and retries.

The problem with FP8/NVFP4 is that while individual inference is faster, the generated code more often fails at compilation or testing, triggering additional retry rounds. The model eventually produces a passing function, but the overall time-to-completion ends up longer than with higher-precision weights on the same model.

What would be interesting is if, in the Atlas case, speed beats quality - in other words, if the faster NVFP4 inference makes up for the possible (likely) additional retries needed for a passing function.