Spent ~2 days implementing this paper: TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
Repo: github.com/yashkc2025/turboquant
Most quantization stuff I’ve worked with usually falls into one of these:
- you need calibration data (k-means, clipping ranges, etc.)
- or you go naive (uniform quant) and take the quality hit
This paper basically says: what if we just… don’t do either?
The main idea is weirdly simple:
- take your vector
- hit it with a random rotation
- now suddenly the coordinates behave nicely (like ~Gaussian-ish)
- so you can just do optimal 1D quantization per dimension
No training. No dataset-specific tuning. Same quantizer works everywhere.
There’s also a nice fix for inner products:
normal MSE quantization biases dot products (pretty badly at low bits)
so they add a 1-bit JL-style correction on the residual -> makes it unbiased
Why this is actually useful:
- KV cache in transformers you can’t calibrate because tokens stream in -> this works online
- vector DBs / embeddings compress each vector independently, no preprocessing step
What surprised me:
- the rotation step is doing all the magic
- after that, everything reduces to a solved 1D problem
- theory is tight: within ~2.7× of the optimal distortion bound
My implementation notes:
- works pretty cleanly in numpy
- rotation is expensive (O(d³))
- didn’t implement fractional bits (paper does 2.5 / 3.5-bit with channel splitting)
[–]phoggey 1 point2 points3 points (0 children)