Speculative Decoding

Speculative decoding accelerates autoregressive generation-where tokens are produced one at a time-by having a lightweight draft model generate K candidate tokens in a single forward pass, then the target large language model (LLM) validates them in parallel. If all K are accepted, the LLM outputs them quickly; if some are rejected, the model regresses to the accepted prefix and re-samples. This technique preserves exact output distribution (no loss in quality) because it uses rejection sampling. Typical speedups range from 2x to 3x on latency-sensitive tasks. The draft model must be much smaller (e.g., 1B parameters for a 70B target) and ideally use the same tokenizer. Popular implementations: Self-Speculative Decoding (using early exit from the same model) and Medusa (adding multiple prediction heads). Key tradeoffs: if draft model accuracy is low, rejection rate increases, negating benefits. Memory overhead for loading both models is minimal if draft is small. Speculative decoding is most effective for tasks where latency per token matters (chat, real-time APIs) and for long sequences where KV cache reuse speeds parallel verification. It works well with batched inference (continuous batching) especially. For AI builders, tuning the draft length K (2-5 tokens) based on your model pair and hardware is critical. Tools like TensorRT-LLM and Hugging Face Text Generation Inference support it.