KV Cache Quantization

KV cache quantization is a technique that reduces the memory footprint of the key-value (KV) cache during autoregressive inference with transformer-based LLMs. In a transformer model, for each token generated, the model computes keys and values from the input tokens, which are stored in memory and reused for subsequent tokens to compute attention. This KV cache grows linearly with the sequence length (both input and generated tokens) and the number of layers and attention heads. For a 7B model with 40 layers, 32 attention heads, a hidden size of 4096, and a context window of 4096 tokens, the KV cache can exceed 10GB in FP16 precision. For models with 70B parameters and 128K context windows, the cache can balloon to hundreds of gigabytes, often exceeding GPU memory. KV cache quantization compresses these stored keys and values from FP16 (2 bytes per value) to lower precision, typically INT8 (1 byte) or FP8 (1 byte), and in some cases INT4 (0.5 byte). This reduction is possible because attention patterns tend to be reliable at low precision. Most inference engines like vLLM, llama.cpp, and TensorRT-LLM support selective quantization of the KV cache, often using per-channel or per-token scaling to minimize accuracy loss. For example, vLLM supports KV cache FP8 quantization on H100 GPUs, achieving up to 2x memory reduction with negligible quality degradation. Some implementations use block-wise quantization (e.g., quantizing in blocks of 128 values) to balance memory and accuracy. The primary benefit is enabling larger batches or longer context windows within the same GPU memory budget. This is critical for applications like long-document analysis, code completion with large repositories, or multi-turn chatbots that need to remember extensive conversation history.