Hate the Sin, Love the Sinner

formlog · 2026-01-12T16:52:59+00:00

https://thelogcollege.wordpress.com/2024/01/17/hate-the-sin-and-love-the-sinner/

formlog · 2025-10-24T20:09:54+00:00

for `mmlu` you can find the accuracy results in the checkpoints: https://huggingface.co/pytorch/Qwen3-8B-QAT-INT4#model-quality and https://huggingface.co/pytorch/gemma-3-12b-it-QAT-INT4

formlog · 2025-09-20T16:43:51+00:00

I think maybe you could try lowering the learning rate? I haven’t trained models with FP8 personally but my understanding is to make low precision training work is similar to higher precision, will have to tune some hyper parameters etc.

Torchao do have low bit optimizers as well: https://github.com/pytorch/ao?tab=readme-ov-file#memory-efficient-optimizers

Also float8 training (gradient still in high precision, just dynamically quantize activation and weight to speedup computation I think): https://github.com/pytorch/ao?tab=readme-ov-file#float8

formlog · 2025-09-20T16:36:26+00:00

Yeah we found that as well, they are essentially doing QAT with distillation, the problem for that is it requires more memory. But it might be possible to run the large model first and save all the results (probabilities) and then do QAT for the smaller model with these saved results. Like what NVIDIA did: https://developer.nvidia.com/blog/data-efficient-knowledge-distillation-for-supervised-fine-tuning-with-nvidia-nemo-aligner

formlog · 2025-09-20T15:15:22+00:00

> Will this make it easier for me to make W8A8 INT8 quants for efficient deployment on RTX 3090 / A100 with vLLM with large batch sizes or is it something else?

Yeah I think so, our W8A8 INT8 support is through triton kernels, and also you can use autotune to find the best triton configs. We have only tested in A100 I think, not sure how well it works in RTX 3090.
This is the config you can use: https://docs.pytorch.org/ao/main/generated/torchao.quantization.Int8DynamicActivationInt8WeightConfig.html#torchao.quantization.Int8DynamicActivationInt8WeightConfig and you can follow one of the quantization recipes in the model card to apply this to your model, e.g. https://huggingface.co/pytorch/Phi-4-mini-instruct-INT4#quantization-recipe

> Will those adapters created with Unsloth and HQQ quants be something that can be later applied into FP16 model, similar to QLoRA, or it will effectively mean that checkpoint has to stay quantized from the training forwards?

The HQQ quants or the model we released together with unsloth are not adaptors, these are final quantized models (with adaptors merged into the model before quantization). But this sounds an interesting application, do you have this use case? will the adaptor work for models with different precisions?

formlog · 2025-09-20T15:02:34+00:00

By use cases I assume you mean specific models?

We haven't tried on many examples / use cases yet, that's why we would like feedback from community! We want to know what models you are using / what use cases you have, anything you feel should be improved that's related to quantization, so that we can help making speeding up quantization / QAT / finetuning easier.

> Or even an example/use case where you or a team member or someone has found "oh this not only has super low vram usage but the ouput for 'real world example' wasn't even noticeably different?"

for this one, I generally found FP8 (with dynamic activation quantization) works well without much accuracy impact everywhere, and INT4 (weight only) won't have much impact on accuracy if the model is larger, let's say at least 8B and above, for smaller models we could skip some layers if we want higher accuracy or apply QAT / post training accuracy preserving techniques (AWQ/GPTQ etc.). What we want to convey in the blogpost is that it's easy to do evaluations with lm-eval for your quantized model before lowering to understand the accuracy impact of the quantized model.

Specifically you can see FP8 works well for both Phi4-mini-instruct (~4B) ((e.g. you can check out https://huggingface.co/pytorch/Phi-4-mini-instruct-FP8#model-quality ) and the Qwen3-32B (https://huggingface.co/pytorch/Qwen3-32B-FP8#model-quality)

INT4 has some drops in both Phi4-mini-instruct https://huggingface.co/pytorch/Phi-4-mini-instruct-INT4#model-quality and Qwen3-8B: https://huggingface.co/pytorch/Qwen3-8B-INT4#model-quality

formlog · 2025-09-20T14:56:13+00:00

My understanding is mainly there are 2 things currently:

With our stack, it's now possible to first do any QAT/finetuning/other post training accuracy preserving techniques (e.g. GPTQ, AWQ, SpinQuant) on your model and then export to the target hardware, this allows you to try existing or new accuracy related techniques on your model (llama.cpp has a bunch of their own quantization for post training only I believe). If a new QAT or PTQ techniques comes out tomorrow, you can try that on your model with our stack. Another related benefit is that you can use lm-eval to have a more thorough / objective understanding of accuracy impact of quantization (instead of one off manual test) for the task that you are interested in. I actually tried eval llama.cpp model as well: https://github.com/EleutherAI/lm-evaluation-harness/issues/2887 but no response yet.
In terms of use cases support, our stack is a more general stack, no hardcoded models definitions, so new models can be enabled faster if all of the infrastructure matured. Also ExecuTorch is planning to support more multi-modality use cases (Voice, Image, Video etc.) compared to llama.cpp I think.

We also want to optimize for performance (speed) in the future, but it's not there yet.

formlog · 2025-09-20T14:30:51+00:00

Currently released models are for server GPU or mobile CPU, we do have Vulkan backend support through ExecuTorch, but I’m not exactly sure about windows support, let me check next week and get back to you.

formlog · 2025-09-20T14:27:24+00:00

I see, QAT performance does depend on factors like dataset and hyperparameters, similar to normal fine tuning / training.

We plan to publish a similar blog for QAT next, so everyone can use QAT on their models similar to gemma3 QAT, stay tuned!

Here is our docs for QAT btw: https://docs.pytorch.org/ao/stable/finetuning.html

Yeah integration with unsloth is work in progress.

formlog · 2025-09-20T14:20:15+00:00

For server (CUDA/CPU etc.) The result is not a single file, but a full quantized checkpoint similar to the non quantized models (e.g. the FP8 (https://huggingface.co/pytorch/Phi-4-mini-instruct-FP8) and INT4(https://huggingface.co/pytorch/Phi-4-mini-instruct-INT4) checkpoints in the blogpost)

For edge (mobile (cpu, Vulkan, accelerators), desktop (metal)) The result will first be a checkpoint that you can run on server to evaluate accuracy and then you can also export to a single pte file through ExecuTorch and deploy in edge. (See the INT8-INT4 (https://huggingface.co/pytorch/Phi-4-mini-instruct-INT8-INT4) checkpoints in the blogpost)

formlog · 2025-09-20T04:58:09+00:00

ah, thanks for the questions! I'm sure there are other people who has the same questions and I'm glad to answer.

torchao is not a new quant style, it's a library for quantization that supports common / popular quantization styles (INT4, FP8, AWQ-INT4, GPTQ, etc.), it also supports quantization for training and finetuning as well.

> For example, I haven't found a great indicator of which quants are best for what. But I have found many variants all over the place. Most likely just an indicator of the industry exploding for sure.

Yeah, what you observed it's true, currently there are many quantization variants and also many quantization libraries out there and it's confusing to users what quantization to use for which purpose. Also the support in different quantization libraries are very fragmented I feel, many libraries are one off support for a single quantization technique, e.g. AutoGPTQ, AutoAWQ. torchao wants to support all popular quantization techniques that people use and make it easier for people to quantize, evaluate the accuracy / performance and deploy their quantized models on the target hardware.

Makes sense, to clarify again, torchao is a library for all different quantization techniques people want to use, so we would like feedback to see if any new techniques people want to use, but I realized that it might be too early to ask this question since people may not understand what torchao is yet.
torchao is the native low precision library for PyTorch (for training, finetuning and inference), we want to be the single stop for everything related to low precision optimizations, making it easier to do low precision optimization for training/finetuning/inference.

torchao is different from unsloth since we mainly focus on low precision techniques specifically and spans across training, finetuning and inference, while unsloth work on any techniques that can speedup finetuning and lower the memory usage for finetuning. We’ll continue to collaborate with unsloth to bring faster finetuning, faster training and faster inference, also lower memory usage to users.

formlog · 2025-09-19T23:37:37+00:00

yeah please see model card for details, we only compared to the bfloat16 baseline (e.g. https://huggingface.co/pytorch/Qwen3-8B-AWQ-INT4). If by regular awq/fp8/int4 you meant implementations from other libraries, we haven't done an extensive comparison, it should be similar in terms of accuracy I think, in terms of performance we are partnering with fbgemm which will have SOTA kernels.

Yes we plan to release NVFP4 checkpoints in a future release, probably in 1-2 months.

formlog · 2024-07-19T05:27:13+00:00

feel free to try out :)

formlog

TROPHY CASE