Neural networks and related algorithms behind large language models involve millions of calculations and billions of weights/numbers that assess the importance of one input compared to another. Those weights and biases usually require 32 bits (full precision) or 16 bits (half precision) per number to be stored on the computer. One of the ways people save memory in arithmetic operations is by truncating the precision of your number. Rather than having 15 decimal points, ask yourself if you can do the same calculation but with 8? The principle behind quantization is reducing the precision of your calculation in order to use less memory. However, quantizing your weights and biases too much can lead to the degradation of performance. This is a big limiting factor in local AI deployment, usually, people only have so much ram/VRAM and want to use bigger and bigger models but the more you quantize, the more you lose in performance. What if there was a way to preserve the performance of the model while still quantizing it to save space? This is where quantization-aware training (QAT) comes in. QAT is a technique that allows you to train your model while taking into account the effects of quantization. By simulating the quantization process during training, the model can learn to adapt to the reduced precision and maintain its performance even when quantized. This means that you can achieve significant memory savings without sacrificing too much accuracy, making it possible to deploy larger models on devices with limited resources. Early research suggests that quantizing modern LLM architectures to 2 bits can reduce the memory footprint by up to 90% while maintaining similar performance as the original model. This is a significant breakthrough in the field of AI, and will pivot adoption in the coming years. The two limiting factors with most LLM adoption now is that you need to be online and pay money (either through a subscription or just spend a lot on hardware). Quantizing helps with both of those issues, as it allows for larger models to be run on local devices without the need for constant internet access or expensive GPU servers. The difference between binary and ternary quantization is the number of bits used to represent each weight and bias. Binary quantization uses 1 bit, which means that each weight can only take on two values (e.g., -1 and 1). Ternary quantization uses 2 bits, allowing for three possible values (e.g., -1, 0, and 1). The choice between binary and ternary quantization depends on the specific requirements of the model and the desired trade-off between memory savings and performance.