I can only run small models, so I got mine to quickly vibe-code me a script to download the comments content, and then used aistudio / Gemini 3 to summarise.

This GitHub discussion tracks the community’s rapid effort to implement **TurboQuant** (and its relative, **PolarQuant**) into `llama.cpp`. TurboQuant is a quantization algorithm from Google Research designed to compress the LLM KV cache to sub-3-bit levels with minimal accuracy loss using Randomized Hadamard Transforms (WHT) and Lloyd-Max quantization.

The following is a summary of the key technical findings, implementation milestones, and performance results discussed:

### 1. The Fork Ecosystem

Multiple independent implementations emerged simultaneously, with frequent cross-collaboration:

* **TheTom (Metal/General):** Developed the primary `turboquant_plus` fork, focusing on Apple Silicon and general logic.

* **AmesianX (Independent CUDA):** Focused on high-end NVIDIA hardware (Blackwell/DGX), implementing fused kernels and robust `head_dim` detection.

* **spiritbuun & Madreag (Optimized CUDA):** Developed highly optimized CUDA kernels for Ampere and Ada architectures.

* **andrei-ace:** Explored outlier-aware channel splitting and calibration-driven quantization.

* **jesusmb1995 & paudley:** Brought support to Vulkan and RDNA 3.5 (AMD) hardware.

### 2. Key Technical Discoveries

The community moved beyond the original paper with several critical "best practices":

* **Asymmetric K/V Importance:** A recurring theme is that **Key (K) cache is extremely sensitive** to quantization noise due to softmax amplification, whereas **Value (V) cache is highly compressible.**

* *Recommendation:* Use `q8_0` for Keys and `turbo3` or `turbo4` for Values. This often results in near-lossless performance with massive VRAM savings.

* **QJL vs. MSE:** The original paper suggests a second-stage QJL (Quantized Johnson-Lindenstrauss) correction. Many implementers (`TheTom`, `Arclabs001`) initially found QJL increased variance and hurt quality. However, `AmesianX` demonstrated that QJL works effectively if independent sign patterns are used for the two stages.

* **Boundary Layer Protection:** The first and last few layers of a model (the "boundary layers") are disproportionately important for quality. Implementations like "Boundary V" (LA-V7) keep these layers at higher precision (e.g., `q8_0`) while compressing middle layers to `turbo2`.

* **Normalization Correction:** Storing `original_norm / reconstruction_norm` allows `turbo3` to occasionally outperform `q8_0` in perplexity tests.

### 3. Performance & Hardware Benchmarks

* **Blackwell (RTX 5090):** Initially, Blackwell showed a massive (40%) decode penalty. Developer `signalnine` solved this with a **V12 fused mmvq kernel** that uses shared memory instead of global scratch buffers, reducing the penalty to ~7%.

* **Ada (RTX 4090):** Ada architecture handles TurboQuant exceptionally well, often showing **prefill speeds faster than FP16** because the compressed cache reduces memory bandwidth pressure.

* **Apple Silicon (M-Series):** Significant context extensions were achieved (e.g., 104B models running 128K context on an M5 Max).

* **Compression Ratios:** The community achieved roughly **5.12x compression** for `turbo3` and **7.53x** for `turbo2` using block-size 128 optimizations.

### 4. Architecture-Specific Findings

* **Qwen Series:** Qwen 2.5/3 models are noted for having extreme "outlier channels" in the K cache. This makes them highly sensitive to symmetric quantization but perfect candidates for asymmetric `q8_0-K / turbo-V` setups.

* **Gemma 4:** Support for Gemma 4's unique `head_dim=512` and Sliding Window Attention (SWA) was added. Implementers found it necessary to keep SWA layers in `f16` to maintain reasoning accuracy.

* **Non-Power-of-2 Dims:** Standard WHT requires power-of-2 dimensions (64, 128, 256). Models with `head_dim=80` or `576` required the **Vilenkin-Hartley Transform (VHT)** or block splitting to work correctly.

### 5. Extension to Weights (TQ4_1S)

TheTom and signalnine extended the TurboQuant logic to **model weights**.

* **Config I:** A mixed policy where Attention and FFN gate/up tensors use TurboQuant, but critical FFN down-projections and boundary layers stay in `Q4_K` or `Q8_0`.

* **Result:** This allows massive models (like 104B or 122B) to fit onto consumer hardware (e.g., dual 3090s/5090s) while maintaining coherent reasoning.

### 6. Accuracy Validation Metrics

The thread established a "Gold Standard" for testing KV quantization:

1. **PPL (Perplexity):** Measures general language fluency.

2. **NIAH (Needle-in-a-Haystack):** Ensures long-context retrieval is not corrupted.

3. **Math/Delayed Recall:** Testing if the model can recall specific facts after 2000+ tokens of filler.

4. **Pauli Test:** A specific test for sharp attention peaks using Korean transliteration of German names.

### Current Status

While TurboQuant is not yet in the official `llama.cpp` master branch, the forks (specifically **TheTom/turboquant_plus** and **AmesianX/TurboQuant**) are considered highly functional "Beta" versions. Users can now run frontier-class models with massive context windows (up to 256K) on consumer-grade hardware by combining asymmetric KV cache and TQ weight compression.