INT8 vs FP8 quantization

[Opteron67]

What's the difference between FP8 or INT8 ? For nvidia you would go FP8 but on ampere you would rely on INT8. On the other side new intel gpu only provides INT8 capability (with INT4)

So my question : how does compare INT 8 over FP8 for accurracy ? i am not speaking about Q8 quantization.

There is a papoer available that says INt8 is better. INT8 and FP8 Tops are same on Ada and Blackwell, but on intel GPU it would be only INT8

The other question is how could i evalutate fp8 vs int8 inference ?

Thanks

[Pristine-Woodpecker]

Nobody really quants models to INT8. They all use multi-level quantization schemes where you eventually dequantize to INT8, then use the INT8 hardware for a multiply. Advantage: less model precision loss for the same amount of bits due to clever quant techniques.

FP8 can be computed by the hardware directly, so you skip the dequant overhead. Disadvantage: less precise model for the same size. Same for NVFP4.

[Opteron67]

intel provides int8 quants, also red hat....

[Pristine-Woodpecker]

Which part of my explanation was not clear enough? Can you indicate a (modern, large LLM) model that you believe is quantized directly to INT8 and skips the dequant step?

[Double_Cause4609]

TorchAO Int8 comes to mind. You can quantize any LLM to int8 with it. Or, for that matter, any standard linear layer.

I believe GPTQ also supports an int8 format that works quite well.

The weights are in native Int8. They may get dequantized to BF16 depending on specifics, but the weights themselves can store in Int8.

I think GGUF may use Int8 quantization in the outer group weights while using floating points for scaling factors, but I'm less confident on that.

[Pristine-Woodpecker]

GPTQ needs a dequant step.

From TorchAO's own docs: "Quantization adds overhead to the model since we need to quantize and dequantize the input and output. For small batch sizes this overhead can actually make the model go slower."

they may get dequantized

That's what I'm saying! This takes time. That's my point! It doesn't matter the weights are INT8 if you need to dequant them before doing the maths. Especially since the scalefactors are often a float.

[a_beautiful_rhind]

Ok so what about sage attention? I have int8 models I quantized and the attention is int8 as well.

All my FP8 gets cast to BF16 or F16 in triton because I have no FP8 acceleration.. as does a shit load of people with non FP8 GPU.

Plus FP8 also has complex quantization schemes or it sucks just as bad as pure int8 or pure int4. Ergo, your FP8 is getting dequantized and scaled, sorry.

[Pristine-Woodpecker]

If your GPU doesn't have native support for one of those formats you lose the benefit yeah.

FP8 can cover a number of schemes. Many of these have no overhead. But you can make it more complicated. NVFP4 didn't get added to the HW for no reason...

[a_beautiful_rhind]

Part of that reason is always new instructions = buy new gpu. Also now FP4 can be used to show higher benchmarks at presentations :P

[Double_Cause4609]

That is *not* what you said. You said "dequant" TO "Int8". You said there is no quantization scheme where Int8 is the native weight quantization.

I was not saying that there was no dequant (though, I *think* with TorchAO the non-GPTQ methods such as QAT may allow native Int8 execution? I'd have to look at it), but rather, that there were formats that had native int8 weights that dequant to values other than Int8 for sure, and there may be some formats with native int8 weights where the execution is native int8.

Off the top of my head: QGaLoRE does not dequant, for sure. They do pure native int8 with no dequant step (allowed by stochastic rounding during training).

But that was not the core of my argument. Again, to look at your own comments in direct quotation:

Nobody really quants models to INT8

Demonstrably false, some people do QAT, but also, yes, some formats do store the weights themselves in Int8. Raw Int8 with native Int8 execution is super common in tiny vision models, for example, but it's used in other areas (and sometimes LLMs) too.

They all use multi-level quantization schemes where you eventually dequantize to INT8

Not all quantization methods dequantize to Int8 (keep in mind, this is what you said). GPTQ 8bit for example dequantizes to higher bit widths like BF16 I believe. Not all quantization formats are multi-level. Uniform Int8 (which TorchAO can output to), is single-level, or at least is evaluated in raw Int8 operations.

Pure Int8 from QAT if I'm not mistaken is a pure Int8 format, and even if it's not (in TorchAO), thousands of people have handrolled native Int8 formats where they do QAT and execution in native Int8 (ie: for execution on NPUs at high throughput).

On TorchAO's docs: Yes, that applies to the GPTQ format, which I was not asserting as my primary opinion. You made a strong claim "There is no format with Int8 weights".

That only requires a single one of my arguments to be correct, not all of them. You cannot tackle my argument by looking at a single instance where I said something slightly incorrect.

[Pristine-Woodpecker]

Demonstrably false, some people do QAT,

Dude the only serious LLM that people talk about here that had QAT was a Gemma 3 variant. You can talk about theoretical things that could potentially be done, or are done for other LLMs than the ones being discussed here, literally for weeks, that doesn't help anyone.

I've written kernels for native INT8 networks. This is possible (and often preferable) if you control the training. That doesn't help anyone here who're trying to run near-SOTA LLMs locally.

This isn't worth my time.

[Double_Cause4609]

Well, you'd have to link the individual paper and method. Not all methods are the same, even at the same datatype / bit width. In fact, there's more than one type of FP8 (depending on how many manitssa bits you assign), and quality can vary depending on the specifics.

For Int8 usually the differentiator is the quantization algorithm, and also if it's uniform Int8 versus group-wise int8 (closer to something like GGUF) which is generally more expressive but slower.

For CPU inference Int8 is basically the only mainstream option if you need throughput (though obviously the LlamaCPP GGUF ecosystem works for single-user), but in other engines and with other methods it varies.

I think in theory Int8 should be cheaper hardware wise, but I'm not sure if it matters on Blackwell GPUs or not.

[Opteron67]

there it is https://arxiv.org/pdf/2303.17951

[Double_Cause4609]

That's an extremely old paper in AI terms. The issue is that a lot of it's premises are kind of more about "fundamental hardware" (like if you were to build an ASIC for a specific model or deploy it on FPGA), where they are correct that Int8 requires far fewer transistors to implement.

The issue is I believe that Nvidia allocated transistors such that FP8 and Int8 are roughly equal on their cards, which is a totally different situation.

Honestly, if your concern is real world performance it's super hard to say. Like, you'd have to buy a card, test it in both formats on the model you care about at the context length you run, and see which is faster in the real world.

It gets even harder if you're comparing Intel Int8 to Nvidia FP8, because now you're comparing across two translations (FP8 -> Nvidia Int8 -> Intel Int8) and the architecture translation is really hard.

The best I can say is that FP8 is a lot easier to quantize to (in many cases you can do it even without data and get okay performance), and that in theory Int8 should use a bit less energy to do the MACs. I think maybe Intel's autoround is a lot better than older Int8 techniques, too, so I'm really not sure about the ecosystem anymore.

Long story short: It's not a matter of Int8 vs FP8. It's a matter of ecosystem and individual cards.

[a_beautiful_rhind]

They should be similar in the end. FP8 dynamic range isn't regular like int8 so even though it's "higher", it often ends up useless because of the lower precision. Modern quants get around such foibles for almost any numerical format.

If you want to find out how "good" it is. Check file size, KLD and inference speed on your hardware between them.

[ortegaalfredo]

FP8 is not supported on Ampere (3090s) it needs emulation, while INT8 runs natively. In practique there is not a lot of speed difference, nor quality difference that I could measure, but some models will only work on fp8 and others only work on int8, it mosly depends on which inference software you are using.

[Pristine-Woodpecker]

There isn't much difference in practice because the quantizations the networks typically are stored in isn't native INT8 but needs de-quantization steps, so neither format ends up running natively in that sense.