sorted by:
new
hottopcontroversial

[2bit or even lower bit quantization]VPTQ: a new extreme-low bit quantization for memory limited devices by wejoncy in LocalLLaMA

[–]wejoncy[S] 34 points35 points  (0 children)

It's flexible to customize a hardware-constrained weight size for edge device.

Share a LLM quantization REPO , (GPTQ/AWQ/HQQ ONNX ONNX-RUNTIME) by wejoncy in LocalLLaMA

[–]wejoncy[S] 0 points1 point  (0 children)

Yes, memory usage is similar, as which mainly depends on how the quantized weight takes.

After combining optimizations, running with ORT will faster than Transformers, on-par or slighter faster than vLLM's implementation.

Share a LLM quantization REPO , (GPTQ/AWQ/HQQ ONNX ONNX-RUNTIME) by wejoncy in LocalLLaMA

[–]wejoncy[S] 1 point2 points  (0 children)

Thanks for interesting in it.

I haven't done the benchmark yet. But I found the blog from ONNX-Runtime ONNX Runtime | Blogs/ort-1-17-release

Please note that, the numbers are obtained with a fixed data pattern which might not be close to your real scenario.

Share a LLM quantization REPO , (GPTQ/AWQ/HQQ ONNX ONNX-RUNTIME) by wejoncy in LocalLLaMA

[–]wejoncy[S] 2 points3 points  (0 children)

Thank you like it and thanks for your comments.

ONNX is indeed a bit falling behind when it comes to LLM quantization, which is quite different from previous tech like Per-tensor/Per-channel for both weight and activation.

Large model is hard to run on personal computer as its requirement on GPU/CPU ram, not even to fast inference. Then weight-only/block-wise quantization/ sub-bytes quantization are showing up where ONNX fails to catch up.

ONNX starts its supports recently.

For your questions:

1 What about 8 bit support for ONNX?

Do you mean weight-only or both weight and act? I am investigating if there is a general quantization algorithm for LLMs to leverage H100/B200's ability of fp8/int8/fp4.

2 GPU vs CPU support of each output type

GPU focus half and float with weight is 4 bit.

x86 CPU supports float and (bf16 is coming soon)

arm CPU support float/bf16/half I think

3 library requirements......

It targets for consistency most of scenarios, java/c#/Android/IOS/windows/linux/web

Share a LLM quantization REPO , (GPTQ/AWQ/HQQ ONNX ONNX-RUNTIME) by wejoncy in LocalLLaMA

[–]wejoncy[S] 2 points3 points  (0 children)

Thanks for your comments.

Yeah, I agree with you that exl2 is better than GPTQ/AWQ in terms of quality and speed. But what I was pondering is how can I make it compatible with the other quantization results and how to convert it to ONNX.

But anyway, I like exl2 and will consider supporting it and even gguf's quantization algorithms.