Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant
Authors: Paolo D'Alberto
Summary
This paper analyzes three quantization schemes for the key-value (KV) cache in transformers under a fixed bit budget: scalar quantization on both K and V (KV baseline), Walsh-Hadamard transform plus quantized Johnson-Lindenstrauss on V only (KQV), and QJL on both K and V (QKQV). The author derives statistical bounds showing that applying QJL to the key matrix K inflates inner-product variance, which softmax then amplifies nonlinearly. Empirically, at the most common bit budget (n=4), KQV consistently outperforms QKQV on KL divergence, geometric K error, and a 6D distance metric. At other budgets (n=2,3,5), QKQV sometimes wins on geometric K reconstruction but always loses on KL divergence, revealing a budget-dependent crossover and a K-V asymmetry.
Main takeaways:
- Three KV cache quantization schemes compared: KV (scalar MSE), KQV (WHT+MSE on K, WHT+MSE+QJL on V), QKQV (QJL on both K and V).
- At n=4 bits (the most practical setting), KQV beats QKQV on every metric—KL divergence, geometric K error, and 6D distance—across all tested distributions.
- The K-V asymmetry is unconditional: quantizing K with QJL consistently hurts KL divergence compared to leaving K as-is.
- A budget-dependent crossover exists: QKQV achieves better geometric K reconstruction at n ∈ {2,3,5}, KQV wins at n ∈ {4,6}.
- KL divergence bridges geometric K error to attention routing corruption and output collapse through softmax's Jensen amplification.
Relevance
Not directly related to my persona or midtraining work—this is about efficient inference via quantization. Included because it's a practical optimization for running large models, which I might care about when scaling up experiments, but it doesn't touch behavioral installation or steering.
Abstract
arXiv:2605.08114v1 Announce Type: new Abstract: We analyse three KV cache quantization schemes under a fair bit budget: \textbf{KV} (scalar MSE baseline), \textbf{KQV} (WHT + MSE on $K$; WHT + MSE + QJL on $V$), and \textbf{QKQV} (WHT + MSE + QJL on both). Starting from the Beta distribution on the hypersphere, we trace how QJL on $K$ inflates inner product variance by $\pi/2$, which softmax amplifies nonlinearly via Jensen's inequality, and we present statistical inference and information metrics to highlight practical differences. Three empirical findings emerge. (1)~At $n=4$ (the practically dominant budget), KQV wins on every measure -- KL divergence, geometric $K$ error, and 6D distance -- across all distributions and ranks tested. (2)~The K--V asymmetry is unconditional: QKQV is consistently worse than KQV in KL divergence at every budget and distribution. (3)~A budget-dependent crossover exists: QKQV achieves better geometric $K$ reconstruction at $n \in {2,3,5}$, KQV at $n \in {4,6}$, invariant to rank and tail weight -- an open rate-distortion problem. $\mathrm{KL}(p_{\mathrm{ref}} | p_{\mathrm{quant}})$, K-only by construction, bridges K direction error to routing corruption and output collapse. We present a sufficient condition when the Jensen mechanism amplifies superlinearly through the softmax. At $n \in {2,3,5}$, QKQV wins geometrically because this assumption does not bind. At $n=4$, elevated K error and KL divergence for QKQV strongly suggest the Jensen mechanism is the operative cause of the crossover, providing a new perspective and explanation.