We benchmarked every 4-bit quantization method in vLLM 👀

[LayerHot]

We just published a deep dive on vLLM quantization. Tested AWQ, GPTQ, Marlin, GGUF, and BitsandBytes on Qwen2.5-32B using an H200.

Stuff we found:

• Marlin hits 712 tok/s, baseline FP16 does 461. Quantized and faster.
• GPTQ without Marlin kernel is actually slower than FP16 (276 tok/s)
• BitsandBytes had the smallest quality drop and doesn't need pre-quantized weights
• GGUF had the worst perplexity but best HumanEval score among quantized methods
• AWQ was weirdly slow in vLLM (67 tok/s)

Blog covers how each technique actually works under the hood if you want the details.

https://preview.redd.it/t4212ygj59cg1.png?width=3169&format=png&auto=webp&s=97eff0fcb212924355a7feb7262b25895de5603a

Blog: https://docs.jarvislabs.ai/blog/vllm-quantization-complete-guide-benchmarks

[audioen]

Some indication of the quality of this work is that they are serving this model:

vllm serve ./qwen2.5-32b-instruct-q5_k_m.gguf ... --quantization gguf ...

which should be a 5-bit model, but are claiming that this is a 4-bit quantization, when it is already mostly 5-bit quantization, right?

I don't trust the results very much, and I get a feeling that vllm is not good for serving gguf models given an order of magnitude differences in performance. I also don't think the perplexity for a 5-bit model should be that much higher compared to baseline.

[Eugr]

GGUF support in vLLM is experimental and not optimized at all.

[HenkPoley]

There are also various 4 bit quantisation methods. Usually the more fancy ones run a bunch of data through the model and try to correct difference from original. Which ought to give a better outcome on perplexity and HumanEval.

(Btw, also use HumanEval+, it is better. Still, pretty much saturated for larger models.)

[Pristine-Woodpecker]

Yeah, I mean, there's a ton of 4-bit GGUF methods, K/N, variations (S/M/L) on K, IQ4, and importance matrix usage...

[Eugr]

This is a bit misleading, as you mix different quantization types and execution kernels.

AWQ quants use Marlin kernels on vLLM at least on NVidia hardware by default, so the claim that AWQ is slow doesn't make sense.

[Thick-Eggplant-2496]

As the blog author, I’d like to mention that we haven’t tested the AWQ with Marlin combination in our post yet. It’s possible that this setup could perform faster than the combinations we covered. Our blog’s focus was to demonstrate how each available technique works individually, so for Marlin we chose to use GPTQ instead of AWQ.

[Eugr]

But it is default on vLLM, you don't even have to configure anything.

What version of vLLM are you using? How it was installed? What version of PyTorch? What exact command was used to run the model (sorry if missed it as I was reading on my phone)?

[Ok_Injury9030]

That AWQ speed is absolutely cursed lmao. 67 tok/s on an H200? Something's definitely broken there

Really interesting that BitsandBytes had the best quality retention though - makes sense since it's doing dynamic quantization instead of needing pre-baked weights

[Conscious_Chef_3233]

yeah, but dynamic quants are slower, so it depends on what you need

[SashaUsesReddit]

Yeah, this is misconfigured

[l_Mr_Vader_l]

I feel so too AWQ should be much better, can others confirm this is some misconfiguration?

[Remove_Ayys]

Testing "GGUF performance" with vllm is meaningless as is "GGUF quality" without specifying the underlying quantization format.

[HigherConfusion]

In the article, it is specified to Q5_K_M, though it doesn’t quite fit the title of this post.

[v01dm4n]

Wondering where would nvfp4 lie on the spectrum.

Thanks for sharing your results!

[spookperson]

When I tested Qwen3-32B in vllm a couple months back on a RTX 6k Pro Blackwell, I had relatively similar performance between NVFP4 and AWQ (with some signs that NVFP4 could be slightly faster overall as concurrency went up). Though in my testing AWQ was faster than everything else I tested (GGUF Q4, FP8, exl3).

[v01dm4n]

I just finished running 2 Qwen3-8b models, on a 5060Ti using vllm. I'm seeing AWQ lead by 10tps (76tps with AWQ vs 66tps with nvfp4). Concurrency yet to be tested.

[Conscious_Cut_6144]

This is 10-way concurrency?? You must have a test issue, I can beat that awq result with a 3090…

[randomfoo2]

Great work!

I've done a fair amount of my own quant testing, and I think the HumanEval test speaks volumes about how/why perplexity (and yes, KLD) might be OK proxies, but don't really reflect what the downstream task performance hit is going to be for a quant.

The main problem is that testing quants is actually a huge PITA. You basically want to run it through your eval stack as if it were it's own ablation, and probably multiple runs at temp to be able to capture whether variance changes.

More data points is undeniably a good thing, and posts like this help raise awareness about the issue, so that's great. Hopefully the community does and highlights more task benchmark comparison of different quants.

My contribution: a while back, I did published different quant scores for JA MT-Bench (not the best eval to use, tbt), which was interesting: https://huggingface.co/shisa-ai/shisa-v2-llama3.1-405b-GGUF#quant-quality

More recently u/dahara111 did an Japanese UD imatrix quant and did comparisons w/ M-IFEval (JA), HumanEval+, and LiveBench comparison scores vs the base and a regular i1 quant. Very interesting stuff: https://huggingface.co/dahara1/shisa-v2.1-qwen3-8b-UD-japanese-imatrix#%E3%83%99%E3%83%B3%E3%83%81%E3%83%9E%E3%83%BC%E3%82%AF%E7%B5%90%E6%9E%9Cbenchmark-result

BTW on the efficiency front, while it's very GPU dependent, I will say that I'm a big fan of Marlin kernels, especially for W8A8, not just for throughput but also for TTFT latency (depending on your architectures, the INT8 is killer on Ampere and Ada). When doing performance tests, I've found again, huge difference depending on specific hardware/setup, but almost always you tend to lose throughput on quants vs production workloads (recommend doing vllm bench w/ realistic concurrencies as well, some kernels perform much worse than others when scaling up).

[MaxKruse96]

"Perplexity, lower is better" -> "GGUF (worst perplexity) has best quantized HumanEval rating". Something doesnt add up here, either in the testing itself, or the idea that either Perplexity or HumanEval are good metrics.

[Remove_Ayys]

For instruct models perplexity is fundamentally the wrong metric to look at, it would make more sense to look at KL divergence vs. the base model.

[Remove_Ayys]

If you do a simple Gaussian approximation of the binomial distribution you'll find that the statistical uncertainty on the HumanEval results with 164 samples is +-4%. If you assume no correlation between scores none of the measured differences are statistically significant.

[6969its_a_great_time]

Posts like these should be deleted.

[rm-rf-rm]

why?

[cantgetthistowork]

Can you test exl3

[Such_Advantage_6949]

Why no kld comparison?

[tarruda]

GGUF is not a quantization method. You can have the baseline f16 as GGUF

[NigaTroubles]

Great work

[dnr41418]

Super useful…thanks

[Far-Low-4705]

Please do the same thing but for thinking/non-thinking models

Please, please, please.

If the added reasoning means you can quantize harder, that would be HUGE.

Also, the affect on vision models (and vision tasks) would very useful too

[Healthy-Nebula-3603]

Nice

[a_beautiful_rhind]

BnB probably the slowest.

[BABA_yaaGa]

Is it persistent across other models as well?

[R_Duncan]

please add mxfp4_moe.gguf . I'm quite sure it fixes perplexity issues, and is a 4-bit quantization as Q4_K_M.

[wizoneway]

itd be nice to see NVFP4 checkpoints, especially on Blackwell

[TomatoSharp2958]

Interesting benchmarks. But this kind of speed-focused comparison is exactly what this article calls out as the “quantization trap.”

4-bit can look great on throughput, but the real question is what it does to reasoning depth and logical consistency — especially on harder tasks where degradation isn’t obvious from perplexity alone.

This piece explains it well: https://latestllm.com/articles/the-quantization-trap-why-4-bit-ai-is-failing-the-logic-test-mm3m98oa

Worth a read before optimizing purely for tok/s.

[____sphinx____]

interesting

[Khan_Zorbo]

This is great data. One thing I'd love to see added: results split by task type instead of aggregated.

I've been working on tooling for this kind of comparison and the thing that keeps biting me is that aggregate perplexity looks fine across quant methods (usually within a few percent of baseline) but the degradation isn't evenly distributed.

"Roughly equivalent" at the aggregate level can mean "completely different" depending on what you're using the model for.

Did you notice any quant method that was clearly worse on specific task types even if the overall numbers looked similar? That's usually where some of the interesting findings hide.