KV Cache

The KV cache (Key-Value cache) is a memory-based optimization essential to efficient transformer inference. During autoregressive text generation, each new token's attention scores must be computed against every previous token in the sequence. Without caching, this would require recomputing the key and value projections for the entire past context on every step-a quadratic cost in sequence length. The KV cache stores the key and value matrices from each earlier time step in GPU memory, so that only the new token's query needs to be projected. This reduces attention computation from O(n^2) to O(n) per token, and the total generation cost becomes roughly O(n^2) over the full sequence (since cache grows linearly). However, the KV cache imposes a significant memory footprint: for a 70B parameter model with 80 layers and 8-head group, the cache can consume over 30 GB for a 4096-token sequence. This often becomes the bottleneck for long-context inference, as it scales linearly with batch size and sequence length. Techniques like PagedAttention (used in vLLM) and sliding window attention manage cache more efficiently by storing it in non-contiguous blocks and evicting stale entries. In quantized inference with ExL2 or GPTQ, the KV cache is typically kept in FP16 to maintain numerical precision, further increasing memory pressure. AI builders must carefully profile KV cache size when selecting GPU hardware-models with high token-to-parameter ratios (e.g., MoE) can be especially demanding. Newer architectures like Mamba attempt to eliminate the KV cache entirely, but at the cost of reduced output quality for certain tasks.