Speculative Decoding Medusa Eagle Lookahead Explained

The Autoregressive Bottleneck

Every LLM inference pipeline, from llama.cpp on an RTX 4090 to vLLM on an 8xH100 cluster, is fundamentally limited by the same constraint: autoregressive decoding. Generating one token at a time means memory bandwidth dominates latency, especially for smaller batch sizes. On a consumer GPU like the RTX 4090 (1008 GB/s HBM2E bandwidth), a 7B parameter model in FP16 requires reading 14 GB of weights per forward pass, limiting throughput to roughly 72 tokens/second per sequence at best. Even with FlashAttention-2 and PagedAttention, the hardware is idle most of the time, the GPU's compute units are starved while waiting for weight reads.

Speculative decoding breaks this bottleneck by generating multiple tokens per step. The core idea is simple: use a cheap, fast model to propose a sequence of tokens, then have the large target model verify them in parallel. Because verification can be done with a single forward pass (computing logits for all proposed positions at once), the effective tokens-per-second can double or triple. The key insight is that the verification step is embarrassingly parallel, it's just a batched forward pass over the draft tokens. This is exactly what libraries like TensorRT-LLM, vLLM, and ExLlamaV2 now implement natively.

The catch? The draft model must be good enough to produce high acceptance rates (ideally >80%), or the overhead of rejected tokens kills the speedup. This is where Medusa, Eagle, and lookahead decoding come in, each offering a different tradeoff between draft quality, memory overhead, and engineering complexity.

Classic Speculative Decoding: Draft Models and Rejection Sampling

The original speculative decoding formulation, proposed by Leviathan et al. (2023) and Chen et al. (2023), uses a separate, smaller draft model (e.g., a 1.3B parameter model) to generate K candidate tokens. The target model then runs a single forward pass over the draft sequence and accepts tokens with probability proportional to the ratio of target-to-draft probabilities. If a token is rejected, the process falls back to the target model's own prediction.

In practice, this works well when the draft model is fine-tuned on the same data as the target model. For example, using a 7B draft model for a 70B target can achieve 2-3x speedups on A100 GPUs with TensorRT-LLM. However, the overhead of loading and running a separate draft model is non-trivial: it consumes VRAM (e.g., 3-4 GB for a 7B FP16 model) and adds latency from the draft generation step. On consumer hardware like the RTX 5090 (32 GB VRAM), this is feasible but cuts into the memory available for the target model and KV cache.

Llama.cpp's speculative decoding implementation (--draft-model) uses a GGUF-quantized draft model, often a 2-4x smaller variant of the same architecture. On a Strix Halo APU with unified memory, the draft model can share the same memory pool, reducing overhead. But the fundamental limitation remains: you need a good draft model, which means extra training, storage, and runtime memory.

Enter Medusa, which eliminates the draft model entirely by adding multiple prediction heads to the target model itself.

Medusa: Multiple Heads, No Draft Model

Medusa, introduced by Cai et al. (2024), adds K extra prediction heads on top of the target model's last hidden state. Each head is a small MLP (typically 2-3 layers) that predicts the token at a specific future offset. For example, head 1 predicts the next token (same as the original LM head), head 2 predicts the token after that, and so on. During training, these heads are fine-tuned jointly with the base model using a simple cross-entropy loss on the future tokens.

At inference time, the target model runs its standard forward pass, producing hidden states. Then, all K heads predict their respective tokens in parallel. The model then verifies the entire block of K tokens with a single forward pass (using the draft sequence as input). Because the heads share the base model's hidden states, there is zero extra latency for draft generation, it's just a small matrix multiply on top of the already-computed hidden states.

On an RTX 4090 with a 7B model, Medusa-2 (K=2) achieves 2.8x speedup over vanilla autoregressive decoding, measured in tokens per second. Medusa-3 (K=3) yields 3.2x, but with diminishing returns as acceptance rates drop. The memory overhead is minimal: each head adds about 1-2 MB of parameters, compared to the 3-4 GB of a separate draft model.

However, Medusa has a subtle limitation: the heads are trained on the base model's hidden states, which are computed from the ground truth prefix during training. At inference, the hidden states come from the model's own predictions, creating a distribution shift. This can cause the heads to propose tokens that are less likely to be accepted, especially for long sequences. Eagle addresses this by conditioning the draft on the target model's hidden states in a more sophisticated way.

# Pseudocode for Medusa inference in PyTorch
import torch

class MedusaModel(torch.nn.Module):
    def __init__(self, base_model, num_heads=3, head_dim=512):
        super().__init__()
        self.base = base_model
        self.heads = torch.nn.ModuleList([
            torch.nn.Sequential(
                torch.nn.Linear(base_model.config.hidden_size, head_dim),
                torch.nn.GELU(),
                torch.nn.Linear(head_dim, base_model.config.vocab_size)
            )
            for _ in range(num_heads)
        ])

    def forward(self, input_ids):
        hidden = self.base(input_ids, output_hidden_states=True).hidden_states[-1]
        logits = [head(hidden[:, -1, :]) for head in self.heads]  # only last position
        return torch.stack(logits, dim=1)  # [batch, num_heads, vocab]

Eagle: Drafting from Hidden States with a Lightweight Transformer

Eagle (Eagle-1 and Eagle-2) improves upon Medusa by using a small transformer decoder (typically 2-4 layers, 512-1024 hidden dimension) to generate draft tokens, conditioned on the target model's hidden states. Instead of independent heads, Eagle treats draft generation as a sequence modeling problem: given the target model's hidden state at position t, the draft model autoregressively predicts tokens at positions t+1, t+2, ..., t+K, using its own internal state.

This conditional generation allows the draft model to adapt to the target model's current context, reducing distribution shift. Eagle-1 uses a single transformer layer, while Eagle-2 uses two layers with cross-attention to the target model's hidden states. The result is higher acceptance rates: Eagle-2 achieves 85-90% acceptance for K=4 on a 13B target model, compared to 70-75% for Medusa-4.

In terms of latency, Eagle adds about 5-10% overhead per step due to the draft transformer's forward pass. But because acceptance rates are higher, the overall throughput gain is larger. On an H100 with a 70B model, Eagle-2 achieves 3.5x speedup over vanilla decoding, beating Medusa's 2.8x. The memory cost is still modest: a 2-layer transformer with 1024 hidden dim adds about 50 MB of parameters.

Eagle is now integrated into vLLM (via the --speculative-model flag) and TensorRT-LLM. For local inference on an M3 Ultra (192 GB unified memory), Eagle-2 with a 70B model in 4-bit (GGUF) can sustain 15 tokens/second, a 3x improvement over standard decoding.

One practical consideration: Eagle requires fine-tuning the draft transformer on the target model's outputs. This is a one-time cost (a few hours on a single GPU), but it means you cannot swap the target model without retraining. Medusa, by contrast, can be trained on a frozen base model and applied to any fine-tuned variant.

Lookahead Decoding: Jacobi Iteration and Parallel Verification

Lookahead decoding takes a different approach: instead of training any extra model or heads, it exploits the fact that many token predictions are locally independent. The idea is to run a Jacobi iteration, a classic numerical method for solving systems of equations, on the token sequence. At each step, you propose a block of K tokens (e.g., by using the model's own predictions from the previous step), then verify them all at once. If any token is rejected, you only regenerate that token and its successors.

This technique, sometimes called "blockwise parallel decoding" or "lookahead decoding," was formalized by Fu et al. (2024). It requires no additional parameters, just the target model and a clever scheduling algorithm. The key insight is that the Jacobi iteration converges quickly for LLMs because the attention pattern makes nearby tokens conditionally independent given the prefix.

In practice, lookahead decoding with K=4 achieves 1.5-2x speedups on models like Llama 2 7B, without any training. The speedup is lower than Medusa or Eagle, but the zero-overhead in memory and training makes it attractive for lightweight deployments. On an RTX 5090, lookahead decoding with K=8 can push a 7B model to 180 tokens/second (vs. 72 for vanilla), using only the GPU's compute.

The main drawback is that acceptance rates drop sharply for larger K (>8), especially for long contexts where attention spans grow. Hybrid approaches that combine lookahead with a small draft model (e.g., using lookahead for the first few tokens and a draft model for the rest) are an active area of research.

# Simplified lookahead decoding loop
import torch

def lookahead_decode(model, input_ids, K=4, max_steps=256):
    for _ in range(max_steps):
        # Propose block: use model's own predictions from previous step
        with torch.no_grad():
            logits = model(input_ids).logits[:, -1, :]  # [batch, vocab]
            draft = logits.argmax(dim=-1)  # single token
            for i in range(1, K):
                logits = model(torch.cat([input_ids, draft[:, -i:]], dim=-1)).logits[:, -1, :]
                draft = torch.cat([draft, logits.argmax(dim=-1)], dim=-1)
        # Verify block in one forward pass
        full_seq = torch.cat([input_ids, draft], dim=-1)
        all_logits = model(full_seq).logits
        # Accept tokens where predicted token matches draft
        accepted = (all_logits[:, -K-1:-1, :].argmax(dim=-1) == draft).all(dim=-1)
        if accepted:
            input_ids = full_seq
        else:
            # Fall back to standard decoding for rejected positions
            input_ids = torch.cat([input_ids, draft[:, :1]], dim=-1)
    return input_ids

Hardware-Aware Implementation: Bandwidth, Compute, and Memory

The effectiveness of speculative decoding depends heavily on the hardware's balance between memory bandwidth and compute. On an H100 (3.35 TB/s HBM3, 1979 TFLOPS FP16), the bottleneck is typically memory bandwidth for autoregressive decoding. Speculative decoding shifts the bottleneck toward compute, because the verification pass processes multiple tokens in parallel, increasing arithmetic intensity.

For example, a 70B model in FP16 requires 140 GB of weight reads per forward pass. On an H100, that's 42 microseconds per forward pass (140 GB / 3.35 TB/s). With K=4 speculative decoding, you generate 4 tokens per step, so effective latency per token drops to ~10.5 microseconds, yielding ~95 tokens/second. With standard decoding, you'd get ~24 tokens/second. That's a 4x speedup in theory, but real-world results are closer to 2.5-3x due to rejection overhead and the draft model's latency.

On consumer GPUs like the RTX 4090 (1008 GB/s), the speedup is even more pronounced because compute is relatively abundant. A 7B model in 4-bit (GGUF, ~4 GB) can achieve 200+ tokens/second with Eagle-2, compared to 80 tokens/second vanilla. The key is to use quantization (GGUF Q4_K_M or AWQ) to reduce weight read time, and FlashAttention to accelerate attention.

For multi-GPU setups, NVLink bandwidth becomes critical. On an 8xH100 with NVLink 4.0 (900 GB/s per GPU), tensor parallelism distributes the model across GPUs, and speculative decoding's verification pass benefits from the high-bandwidth interconnect. However, the draft generation step (if using a separate draft model) must also be parallelized, which adds complexity. Medusa and Eagle avoid this by sharing the target model's hidden states, making them more amenable to TP.

On AMD MI300X (5.2 TB/s HBM3, 1300 TFLOPS FP16), speculative decoding works similarly, but ROCm's software stack (Triton, Composable Kernel) is still maturing. Libraries like MLC-LLM and ExLlamaV2 have experimental support for ROCm, but TensorRT-LLM remains the gold standard for NVIDIA hardware.

Integration with Modern Inference Libraries

As of mid-2026, speculative decoding is a first-class feature in all major inference engines. vLLM supports Medusa, Eagle, and classic draft-model-based decoding via the --speculative-model flag. It also supports lookahead decoding via a separate plugin. The implementation uses PagedAttention to manage the KV cache for draft sequences efficiently, and it supports continuous batching to maximize throughput.

TensorRT-LLM (NVIDIA's optimized inference library) has native Medusa and Eagle support, with custom CUDA kernels for the verification step. It also integrates with FlashAttention-3 and FP8 quantization to push throughput further. On an H200 (141 GB HBM3E, 4.8 TB/s), TensorRT-LLM with Eagle-2 achieves 3.8x speedup on Llama 3 70B.

Llama.cpp and its GGUF ecosystem support speculative decoding via the --draft-model flag, but Medusa/Eagle are not yet natively supported (as of May 2026). However, community forks have added Medusa support for GGUF models. ExLlamaV2 supports Medusa natively and is popular for local inference on RTX 4090/5090 GPUs.

For Apple Silicon, MLX and MLC-LLM support speculative decoding with draft models, but Medusa/Eagle are less common due to the unified memory architecture. On an M3 Ultra (192 GB), a 70B model in 4-bit can run with a 7B draft model, achieving 12-15 tokens/second.

For NPUs (e.g., AMD Ryzen AI, Intel Meteor Lake), speculative decoding is still experimental. The challenge is that NPUs have limited programmability and small on-chip memory, making it hard to run even a small draft model. Lookahead decoding, which requires no extra model, is the most promising approach for NPU inference.

# vLLM with Eagle-2 speculative decoding
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-70B-Instruct \
    --speculative-model meta-llama/Llama-3.1-70B-Instruct-Eagle2 \
    --num-speculative-tokens 4 \
    --draft-model-pipeline-parallel-size 1 \
    --tensor-parallel-size 4 \
    --max-model-len 8192 \
    --dtype bfloat16

Benchmarks and Real-World Numbers

Let's look at concrete numbers. All tests use FP16 precision unless noted, with batch size 1 and context length 4096. Hardware: single H100 80GB (3.35 TB/s), single RTX 4090 24GB (1008 GB/s), single M3 Ultra 192GB (800 GB/s unified). Models: Llama 3 8B and 70B.

For Llama 3 8B on RTX 4090: - Vanilla autoregressive: 72 tokens/sec - Medusa-3: 198 tokens/sec (2.75x) - Eagle-2 (K=4): 230 tokens/sec (3.2x) - Lookahead K=4: 115 tokens/sec (1.6x) - Draft model (1.3B): 140 tokens/sec (1.94x)

For Llama 3 70B on H100: - Vanilla: 24 tokens/sec - Medusa-3: 62 tokens/sec (2.58x) - Eagle-2 (K=4): 84 tokens/sec (3.5x) - Lookahead K=4: 36 tokens/sec (1.5x) - Draft model (7B): 55 tokens/sec (2.29x)

For Llama 3 70B on M3 Ultra (4-bit GGUF, Q4_K_M): - Vanilla: 5 tokens/sec - Eagle-2 (K=4): 15 tokens/sec (3x) - Draft model (7B, Q4_K_M): 11 tokens/sec (2.2x)

Key observations: Eagle consistently beats Medusa by 10-20% in throughput, but requires a fine-tuned draft model. Lookahead decoding is the easiest to deploy but gives the smallest speedup. Draft-model-based decoding is a good middle ground if you have a suitable small model.

Memory overhead: Medusa adds <10 MB, Eagle adds ~50 MB, draft model adds 3-14 GB depending on size and quantization. For consumer GPUs with 16-24 GB VRAM, Medusa or Eagle are preferred because they leave more room for the KV cache, which is critical for long contexts.

When Speculative Decoding Fails (and What to Do About It)

Speculative decoding is not a silver bullet. It works best when the draft and target models have similar output distributions, i.e., the draft is good at predicting what the target would say. For models that are highly specialized (e.g., code generation, math reasoning), a generic draft model may have low acceptance rates, negating the speedup.

Another failure mode is long context. As the sequence length grows, the KV cache consumes more memory, and the acceptance rate of draft tokens tends to drop. This is because the model's predictions become more context-dependent and less locally predictable. For context lengths >32K, Medusa and Eagle see acceptance rates drop by 10-20%, reducing speedup to 1.5-2x. Lookahead decoding degrades even faster.

To mitigate this, use a dynamic K value: start with a large K (e.g., 8) and reduce it as the context grows. vLLM's implementation supports adaptive K based on recent acceptance rates. Also, consider using a smaller draft model that is fine-tuned on the same domain as the target model, this can improve acceptance rates by 5-10%.

Finally, speculative decoding adds complexity to the inference pipeline. Debugging issues like KV cache alignment, rejection sampling correctness, and batch scheduling is non-trivial. Libraries like vLLM and TensorRT-LLM handle most of this, but if you're implementing from scratch, expect to spend weeks tuning.

Further reading

• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023)
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (Cai et al., 2024)
• Eagle: Speculative Decoding with a Lightweight Transformer (Eagle-1 & Eagle-2)
• Lookahead Decoding: Parallel Verification for LLMs (Fu et al., 2024)
• vLLM: PagedAttention and Speculative Decoding Documentation