Why tokens per second drops with longer context and how to fix it

The quadratic wall: why longer contexts crush throughput

Every transformer-based LLM relies on the attention mechanism, which computes a similarity score between every pair of tokens in the sequence. For a context of length N, the standard dot-product attention requires O(N^2) floating-point operations and, critically, O(N^2) memory accesses to the attention matrix. This is not just a theoretical curiosity: on real hardware, the memory bandwidth bottleneck dominates. On an H100 with 3.35 TB/s HBM3 bandwidth, the attention step for a 4096-token sequence takes about 0.5 ms; for 32K tokens, it balloons to over 32 ms per layer. Multiply by 80 layers for a 70B model, and you get 2.56 seconds per single forward pass. That is before any autoregressive generation. The token generation phase is even worse: each new token recomputes attention against the entire past sequence, so the cost per token grows linearly with context length. The result is that tokens per second (tok/s) can drop by 5x-10x when moving from 2K to 32K context. This is not a bug; it is the mathematics of quadratic scaling colliding with finite memory bandwidth.

# Rough cost model for attention on H100 (80 layers, 70B model)
# Assume 3.35 TB/s HBM, 312 TFLOPS FP16
# For N=4096: attention FLOPs = 2*N^2*d_model = 2*4096^2*8192 ≈ 2.75e11 FLOPs
# Time = max(FLOPs/TFLOPS, bytes/bandwidth)
# With KV cache: 2*N*d_model*2 bytes = 2*4096*8192*2 = 134 MB per layer
# 80 layers => 10.7 GB, bandwidth limited: 10.7 GB / 3.35 TB/s = 3.2 ms
# For N=32768: 2*32768^2*8192 = 1.76e13 FLOPs, 56.3 ms compute limited
# KV cache: 2*32768*8192*2 = 1.07 GB per layer, 85.6 GB total => 25.6 ms bandwidth limited
# Total per token: ~82 ms => ~12 tok/s, vs 4096: ~3.2 ms => 312 tok/s

KV cache: the hidden memory tax that kills batch size

During autoregressive generation, the transformer caches the Key and Value tensors from every previous token to avoid recomputing them. For a 70B model with 80 layers, d_model=8192, and 32 attention heads, each token consumes 2 * 80 * 8192 * 2 bytes = 2.6 MB of KV cache in FP16. At 32K context, that is 85 GB for a single sequence. An H100 has 80 GB of HBM3, so you cannot even fit one full sequence with standard FP16 KV cache. To run a batch of 4 sequences, you need 340 GB, which forces offloading to CPU memory via PCIe Gen5 (64 GB/s) or NVLink (900 GB/s). The bandwidth penalty is catastrophic: offloading just 1 GB over PCIe costs ~15 ms, and the generation loop cannot hide that latency. The practical result is that batch size must shrink to 1 or 2 for long contexts, which directly reduces throughput. For example, on an RTX 4090 (24 GB, 1.0 TB/s), a 7B model with 8K context can batch 8 sequences at 40 tok/s; at 32K context, batch size drops to 1 and tok/s falls to 12. This is why PagedAttention (vLLM) and KV cache quantization are not optional, they are mandatory for any production system serving long contexts.

# KV cache size for LLaMA-70B (80 layers, 8192 dim, 32 heads, FP16)
# Per token: 2 * 80 * 8192 * 2 bytes = 2,621,440 bytes ≈ 2.6 MB
# For N=32768: 2.6 MB * 32768 = 85,000 MB ≈ 83 GB
# H100 VRAM: 80 GB -> cannot fit one sequence without compression
# With KV cache quantization to 8-bit: 42 GB -> fits one sequence
# With 4-bit: 21 GB -> can batch 3-4 sequences

FlashAttention: tiling away the quadratic memory bottleneck

FlashAttention (Dao et al., 2022) is the single most impactful algorithmic mitigation for long-context throughput collapse. Instead of materializing the full NxN attention matrix in HBM, it tiles the computation over the on-chip SRAM (shared memory on GPUs, typically 192 KB per SM on H100). The key insight is that the softmax normalization can be computed incrementally using a running rescaling trick, avoiding the need to store all intermediate values. This reduces the HBM reads/writes from O(N^2) to O(N^2 / M) where M is the SRAM size. For N=32K and M=192 KB, the reduction is roughly 170x. In practice, FlashAttention-2 on an H100 achieves about 2x-3x speedup over a naive PyTorch implementation for long sequences, and crucially, it eliminates the O(N^2) memory footprint, allowing much larger contexts to fit in VRAM. FlashAttention-3, announced in 2024, adds BF16 and FP8 support with hardware-specific optimizations for Hopper GPUs, further reducing latency. However, FlashAttention does not reduce the FLOP count; it only reduces memory traffic. The compute-bound quadratic FLOPs remain, so for very long contexts (e.g., 128K), even FlashAttention cannot prevent throughput collapse. That is where sparsity and approximation come in.

PagedAttention and vLLM: virtual memory for the KV cache

PagedAttention, introduced in the vLLM system (Kwon et al., 2023), addresses the KV cache memory fragmentation problem. In standard implementations, the KV cache is allocated as a contiguous block per sequence. When sequences have different lengths or when new sequences start, this leads to severe internal and external fragmentation. PagedAttention treats the KV cache as a set of fixed-size pages (typically 16 tokens per page) and maps them via a page table, similar to virtual memory in an OS. This allows the system to allocate only the pages needed, reuse pages across sequences for shared prefixes (e.g., system prompt), and efficiently handle preemption and swapping. The throughput gains are dramatic: vLLM achieves up to 2x-4x higher throughput than naive implementations on the same hardware, especially under mixed-length workloads. For long-context scenarios, PagedAttention reduces the effective memory footprint by eliminating fragmentation and enabling larger batch sizes. Combined with FlashAttention, vLLM is the de facto standard for production LLM inference. On an 8xH100 node, vLLM with PagedAttention can serve a 70B model at 32K context with 8 sequences in batch, achieving ~30 tok/s per user, compared to <10 tok/s without.

# vLLM configuration for long-context 70B on 8xH100
model: meta-llama/Llama-2-70b-chat-hf
tensor_parallel_size: 8
max_model_len: 32768
gpu_memory_utilization: 0.95
block_size: 16
swap_space: 8
# PagedAttention block size 16 reduces fragmentation
# With 95% VRAM usage, ~76 GB per GPU, 8 GPUs => 608 GB total
# KV cache for 32K context per sequence: ~85 GB
# Can batch ~7 sequences concurrently

KV cache compression: quantize, prune, slide

Beyond paging, compressing the KV cache directly reduces memory pressure and bandwidth. The three main approaches are quantization, pruning, and sliding windows. Quantization: converting the KV cache from FP16 to 8-bit or 4-bit reduces memory by 2x-4x with minimal accuracy loss. Techniques like KIVI (Liu et al., 2024) and KVQuant (Zhao et al., 2024) show that the KV cache is highly amenable to low-bit quantization because the values are typically small and have low variance. On an H100, using 8-bit KV cache for a 70B model at 32K context drops the cache from 85 GB to 42.5 GB, allowing two sequences to fit. Pruning: attention patterns are often sparse, especially in later layers. StreamingLLM (Xiao et al., 2024) and H2O (Heavy Hitter Oracle, Zhang et al., 2024) show that you can discard up to 90% of the KV cache tokens without significant perplexity degradation by keeping only the most attended tokens. Sliding window attention (Mistral, 2023) limits the context to a fixed window (e.g., 4096 tokens), so the KV cache size is bounded. This is the simplest mitigation but limits the model's ability to attend to distant tokens. For applications like RAG where the relevant context is usually within a few thousand tokens, sliding window is often sufficient.

Multi-query and grouped-query attention: reducing the KV cache per head

Standard multi-head attention (MHA) uses a separate Key and Value head for each Query head. For a 70B model with 32 heads, that is 32 KV heads per layer. Multi-query attention (MQA) uses a single KV head shared across all Query heads, reducing the KV cache size by a factor of 32. Grouped-query attention (GQA) is a compromise, using 4 or 8 KV heads (e.g., LLaMA-2-70B uses 8 KV heads, a 4x reduction). This directly translates to 4x less KV cache memory and bandwidth. On an H100, a GQA-based 70B model at 32K context has a KV cache of ~21 GB (8-bit), allowing 3-4 sequences to fit in 80 GB. The compute cost of attention still scales quadratically with N, but the memory bottleneck is significantly relaxed. GQA is now standard in most modern LLMs (LLaMA-2, Mistral, Mixtral, Gemma). If you are deploying a model that still uses MHA (e.g., older GPT variants), consider fine-tuning with GQA or using a model that already has it. The throughput gain is often 2x-3x for long contexts.

# KV cache memory comparison for 70B model, 32K context, FP16
# MHA (32 KV heads): 2 * 80 * 8192 * 32 * 2 bytes = 83.9 GB
# GQA (8 KV heads): 2 * 80 * 8192 * 8 * 2 bytes = 20.97 GB
# MQA (1 KV head): 2 * 80 * 8192 * 1 * 2 bytes = 2.62 GB
# With 8-bit quantization: divide by 2 for each

Speculative decoding: generating multiple tokens per forward pass

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) is a technique that uses a small, fast draft model to generate a sequence of candidate tokens, which are then verified by the large target model in a single forward pass. Because verification can be done in parallel (the target model processes all candidate tokens simultaneously), the effective cost per token drops. For long contexts, the draft model's KV cache is much smaller (e.g., a 7B draft vs a 70B target), so the memory overhead of the draft is negligible. The target model still pays the quadratic attention cost for the full context, but it only does so once per block of k tokens, rather than once per token. Typical acceptance rates are 0.7-0.9, yielding 2x-3x speedups. On an 8xH100 system with a 70B target and 7B draft, speculative decoding can boost throughput from 30 tok/s to 70 tok/s for 32K context. The catch is that the draft model must be aligned with the target (same tokenizer, similar distribution), which requires careful training or fine-tuning. Libraries like Medusa (Cai et al., 2024) and Eagle (Li et al., 2024) extend this idea with multiple draft heads and tree-based verification, achieving up to 4x speedups.

Hardware realities: HBM bandwidth, NVLink, and PCIe bottlenecks

No amount of algorithmic optimization can fully escape the physical limits of memory bandwidth. The H100's 3.35 TB/s HBM3 is already the fastest commercially available, but it is still orders of magnitude slower than the compute throughput. For context, the H100's 312 TFLOPS (FP16) can theoretically perform 312e12 operations per second, but each operation requires data from HBM. The ratio of compute to bandwidth is about 93 FLOPs per byte. For attention, the arithmetic intensity (FLOPs per byte) is low: for N=4096, it is about 10 FLOPs per byte; for N=32768, it drops to 1.25 FLOPs per byte. This means attention is almost always bandwidth-bound. The only way to improve throughput is to increase effective bandwidth or reduce memory traffic. NVLink (900 GB/s per GPU on H100) and NVSwitch (7.2 TB/s aggregate) help by allowing KV cache sharding across GPUs via tensor parallelism. For example, with 8-way tensor parallelism, each GPU holds 1/8 of the KV cache, reducing per-GPU memory pressure and bandwidth demand. However, inter-GPU communication adds latency. The sweet spot for long-context inference is often 4-way or 8-way tensor parallelism with FlashAttention and PagedAttention. On AMD MI300X (5.2 TB/s HBM3, 896 GB/s Infinity Fabric), the same principles apply, but the software stack (ROCm, Triton) is less mature, so real-world throughput may be 20-30% lower than H100 for the same model.

Practical deployment: a recipe for 32K context at 30+ tok/s on 8xH100

Combining all the mitigations, here is a concrete recipe for running a 70B model at 32K context with high throughput. Use vLLM with PagedAttention (block size 16), FlashAttention-2, 8-way tensor parallelism, KV cache quantization to 8-bit, and speculative decoding with a 7B draft model. Set max_num_seqs to 8 and gpu_memory_utilization to 0.95. On 8xH100, this configuration achieves about 30 tok/s per user with 8 concurrent users, or 240 tok/s aggregate. Without these mitigations, the same setup would struggle to serve even 2 users at 10 tok/s. The key is that each mitigation attacks a different bottleneck: PagedAttention reduces fragmentation, FlashAttention reduces memory traffic, KV quantization halves the cache size, GQA (already in the model) reduces per-head cache, and speculative decoding reduces the number of forward passes. The combined effect is multiplicative. For even longer contexts (128K), you need additional techniques like sliding window attention or sparse attention, but those come with accuracy trade-offs. For most RAG and chat applications, 32K context is sufficient, and this recipe provides a production-ready baseline.

# Example vLLM launch command for 8xH100, 70B, 32K context
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-2-70b-chat-hf \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --block-size 16 \
  --kv-cache-dtype fp8 \
  --speculative-model meta-llama/Llama-2-7b-chat-hf \
  --num-speculative-tokens 5 \
  --max-num-seqs 8 \
  --enable-chunked-prefill

The future: sparse attention, hardware accelerators, and NPUs

The next frontier for long-context inference is sparse attention, where the model learns to attend only to a subset of tokens. Techniques like Reformer (Kitaev et al., 2020), Longformer (Beltagy et al., 2020), and BigBird (Zaheer et al., 2020) use fixed or learned sparsity patterns to reduce the O(N^2) cost to O(N log N) or O(N). However, these require retraining the model, which is impractical for existing LLMs. More promising are hardware accelerators like the NVIDIA B200 (Blackwell) with 192 GB HBM3e and 8 TB/s bandwidth, which doubles the memory capacity and bandwidth of H100. The AMD MI400, expected in 2026, may offer similar improvements. For edge devices, NPUs like Apple's Neural Engine (M3 Ultra) and AMD's Strix Halo (XDNA 2) are adding on-chip memory for KV cache, but their bandwidth is limited (e.g., M3 Ultra has 800 GB/s unified memory). On these platforms, extreme quantization (4-bit KV cache) and small models (7B-13B) are the only viable path for long contexts. MLX and MLC-LLM are the leading frameworks for Apple Silicon, but they still lag behind CUDA in FlashAttention support. The long-term solution may be custom ASICs with in-memory computing or analog attention, but that is years away. For now, the best strategy is to combine all available algorithmic mitigations and choose hardware with the highest memory bandwidth per dollar.

Further reading

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• PagedAttention: Efficient Memory Management for Large Language Model Serving
• KIVI: A Tuning-Free Asymmetric 2-bit Quantization for KV Cache
• Speculative Decoding: Fast Generation from Large Language Models
• vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention