- LLM inference is memory-bandwidth-bound for batch size 1 and low batch sizes; compute becomes the bottleneck only at high batch sizes.
- HBM3E on H200 delivers 4.8 TB/s, a 1.6x improvement over H100's 3.35 TB/s, directly translating to ~1.6x higher tokens/sec for memory-bound workloads.
- HBM3E's 24 GB stacks (vs 16 GB in HBM3) enable larger capacity per die, reducing the need for multi-GPU sharding for models like 70B and 405B.
- Bandwidth scaling with HBM generations follows a power law: each new generation roughly doubles bandwidth per stack, but thermal and packaging constraints limit practical gains.
- For AI builders, optimizing for HBM bandwidth means using quantization (FP8, INT4), flash attention variants, and efficient kernel fusion to keep the memory pipeline saturated.
The Bandwidth Wall: Why Compute Is No Longer the Limit
For years, the AI hardware narrative centered on FLOPS. NVIDIA touted H100's 1979 TFLOPS in FP8, AMD claimed MI300X's 1307 TFLOPS, and the industry chased ever-higher compute density. But for large language model inference, especially autoregressive decoding, compute is rarely the bottleneck. The real constraint is memory bandwidth. Each generated token requires the entire model to be read from HBM into the compute units, processed through attention and feed-forward layers, and written back. At batch size 1, every token generation is a memory-bound operation: the time to read the model weights dominates the time to compute. For a 70B parameter model in FP16, that's 140 GB of weights. On an H100 with 3.35 TB/s HBM3 bandwidth, the theoretical minimum time to read all weights is 140 GB / 3.35 TB/s = 41.8 microseconds. In practice, with overhead, you get about 20-30 tokens per second for a 70B model. Double the bandwidth to 4.8 TB/s (H200 HBM3E), and you get 30-40 tokens per second. This simple arithmetic explains why HBM bandwidth is the single most important spec for LLM inference, often overshadowing raw TFLOPS.
# Bandwidth-limited token generation estimate
model_params = 70e9 # 70B
bytes_per_param = 2 # FP16
weight_size_gb = model_params * bytes_per_param / 1e9 # 140 GB
# H100 HBM3 bandwidth: 3.35 TB/s = 3350 GB/s
h100_bw = 3350 # GB/s
time_per_token_h100 = weight_size_gb / h100_bw # ~0.0418 s = 41.8 ms
tokens_per_sec_h100 = 1 / time_per_token_h100 # ~23.9 tok/s
# H200 HBM3E bandwidth: 4.8 TB/s = 4800 GB/s
h200_bw = 4800
time_per_token_h200 = weight_size_gb / h200_bw # ~0.0292 s = 29.2 ms
tokens_per_sec_h200 = 1 / time_per_token_h200 # ~34.3 tok/s
print(f"H100: {tokens_per_sec_h100:.1f} tok/s, H200: {tokens_per_sec_h200:.1f} tok/s")These estimates assume perfect bandwidth utilization, which is never achieved. Real-world vLLM or TensorRT-LLM benchmarks show 20-25 tok/s on H100 and 30-35 tok/s on H200 for Llama 3 70B.
HBM3 vs HBM3E: The Technical Differences
HBM3 (High Bandwidth Memory 3) was standardized by JEDEC in 2022, offering per-pin data rates up to 6.4 Gbps, with 8 to 12 dies per stack and 16 GB per stack. Each HBM3 stack has a 1024-bit wide interface, yielding peak bandwidth per stack of 819 GB/s at 6.4 Gbps. The H100 SXM uses 6 stacks of HBM3, totaling 96 GB capacity and 3.35 TB/s bandwidth (6 * 819 GB/s * 0.68 efficiency factor due to memory controller overhead and clocking). HBM3E, an enhanced version finalized in 2024, pushes per-pin data rates to 9.2 Gbps and increases per-die capacity from 16 Gb to 24 Gb, enabling 24 GB per stack. H200 uses 6 stacks of HBM3E, delivering 141 GB capacity and 4.8 TB/s bandwidth. The 1.6x bandwidth improvement directly translates to 1.6x tokens per second in memory-bound scenarios. But the real win is capacity: 141 GB vs 96 GB means you can fit a 70B FP16 model (140 GB) on a single H200, whereas on H100 you need two GPUs or quantization. This eliminates the overhead of tensor parallelism across GPUs for that model size, reducing latency and complexity.
When choosing between H100 and H200 for inference, the capacity boost is often more impactful than bandwidth. A single H200 can serve 70B models in FP16 without sharding, simplifying deployment.
Real-World Impact: Tokens per Second Across Hardware
Let's ground the theory with concrete numbers. Using vLLM 0.6.0 with Llama 3 70B in FP16, batch size 1, input length 128, output length 256: on H100 SXM (3.35 TB/s), you get approximately 22 tokens/second. On H200 (4.8 TB/s), the same setup yields 34 tokens/second. On AMD MI300X (5.2 TB/s), you get about 37 tokens/second, though software maturity with ROCm and vLLM still trails NVIDIA. For smaller models like Llama 3 8B, the bandwidth advantage is less pronounced because the weights fit in L2 cache or are small enough that compute starts to matter. At batch size 64, the bottleneck shifts to compute: H100's 1979 TFLOPS in FP8 becomes the limiter, and bandwidth differences shrink. The key insight: for interactive applications (batch size 1-4), bandwidth is king. For high-throughput serving (batch size 64+), compute and memory bandwidth both matter, but HBM3E still helps by reducing the time to load weights for each batch.
# Example vLLM benchmark command for H200
python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size 1 --dtype float16 --max-model-len 4096 \
--gpu-memory-utilization 0.95 --enforce-eager
# Then run benchmark:
python benchmarks/benchmark_throughput.py --model meta-llama/Meta-Llama-3-70B-Instruct \
--backend vllm --num-prompts 100 --input-len 128 --output-len 256Bandwidth Scaling and the Memory Wall: A Historical Perspective
Memory bandwidth has historically grown slower than compute. From HBM2 (2 TB/s on V100) to HBM2E (2.4 TB/s on A100) to HBM3 (3.35 TB/s on H100) to HBM3E (4.8 TB/s on H200), the generational improvement is roughly 1.5x per generation. Meanwhile, compute FLOPS have grown 2-3x per generation (V100 125 TFLOPS FP16, A100 312 TFLOPS, H100 1979 TFLOPS FP8). This widening gap means that for many workloads, we are increasingly memory-bound. The 'memory wall' is not a future problem; it is the present reality for LLM inference. Future HBM4, expected in 2026, promises per-stack bandwidth of 1.6 TB/s (up from HBM3E's 1.2 TB/s) and capacities up to 64 GB per stack. But even that will only temporarily relieve the bottleneck. The industry is responding with techniques like quantization (FP8, INT4, INT8), flash attention to reduce memory reads, and speculative decoding to reduce the number of memory-bound steps. But for the foreseeable future, HBM bandwidth will remain the primary lever for inference performance.
Don't be fooled by peak TFLOPS numbers. For LLM inference at low batch sizes, HBM bandwidth is 10x more important than compute. Always check the bandwidth spec first.
The Role of Quantization: Stretching Bandwidth Further
Quantization reduces the number of bytes per parameter, effectively increasing the 'effective bandwidth' of your memory system. A 70B model in FP16 requires 140 GB; in INT4, it's 35 GB. With HBM3E's 4.8 TB/s, reading 35 GB takes 7.3 microseconds, yielding a theoretical 137 tokens/second. In practice, quantization adds compute overhead (dequantization, scaling), and the model's accuracy may degrade. AWQ and GPTQ are popular weight-only quantization methods that preserve accuracy at INT4. For inference, ExLlamaV2 and llama.cpp with GGUF format support INT4 and even 2-bit quantization (e.g., IQ2_XXS) that can fit a 70B model in 20 GB. However, quantization also increases the number of memory accesses per parameter (due to packing and scaling factors), so the bandwidth benefit is not perfectly linear. On H200, a 70B model in INT4 can achieve over 100 tokens/second with ExLlamaV2, compared to 34 tokens/second in FP16. This is why serious AI builders always pair high-bandwidth memory with aggressive quantization.
# Effective bandwidth with quantization
model_params = 70e9
bytes_per_param_fp16 = 2
bytes_per_param_int4 = 0.5 # 4 bits = 0.5 bytes
bw_h200 = 4800 # GB/s
# FP16
time_fp16 = (model_params * bytes_per_param_fp16 / 1e9) / bw_h100 # 0.0292 s
tok_per_sec_fp16 = 1 / time_fp16 # 34.3
# INT4
time_int4 = (model_params * bytes_per_param_int4 / 1e9) / bw_h200 # 0.0073 s
tok_per_sec_int4 = 1 / time_int4 # 137.0
print(f"FP16: {tok_per_sec_fp16:.1f} tok/s, INT4: {tok_per_sec_int4:.1f} tok/s")Software Optimization: Making Every Byte Count
Bandwidth is only useful if your software can keep the memory pipeline full. vLLM's PagedAttention, FlashAttention-2, and TensorRT-LLM's in-flight batching are all designed to maximize memory bandwidth utilization. PagedAttention eliminates memory fragmentation in the KV cache, allowing larger batch sizes that better saturate bandwidth. FlashAttention-2 tiles the attention computation to reduce HBM reads/writes, effectively increasing the fraction of bandwidth used for weight loading rather than intermediate data. TensorRT-LLM fuses multiple operations (layernorm, activation, matrix multiply) into single kernels, reducing the number of kernel launches and memory round-trips. On H200, these optimizations can push bandwidth utilization from 60% to 85%+, directly translating to higher tokens per second. For example, with TensorRT-LLM and FP8 quantization, a 70B model on H200 can achieve over 50 tokens/second at batch size 1, compared to 34 tokens/second with naive PyTorch. The lesson: hardware bandwidth is the ceiling, but software determines how close you get.
Using FlashAttention-2 with vLLM on H200 can increase throughput by 15-20% for long-context models due to reduced memory traffic.
Comparing HBM3E Across Vendors: NVIDIA, AMD, Intel, and Apple
NVIDIA's H200 SXM leads with 4.8 TB/s, but AMD's MI300X offers 5.2 TB/s (8 stacks of HBM3 at 6.5 Gbps). However, AMD's software stack (ROCm, vLLM support, TensorRT-LLM alternative) is less mature, so real-world performance often lags behind the raw bandwidth number. Intel's Gaudi 3 uses 8 stacks of HBM2E at 3.7 TB/s, which is competitive but not class-leading. Apple's M3 Ultra uses a unified memory architecture with 800 GB/s, which is far below HBM3E but benefits from no PCIe overhead and tight integration with MLX. For on-premise deployments, the choice is not just about bandwidth but about the entire ecosystem. For example, running a 405B model (FP16, 810 GB) requires at least 6 H200 GPUs (6 * 141 GB = 846 GB) with NVLink for fast communication. On MI300X, you'd need 8 GPUs (8 * 192 GB = 1536 GB) but with slower Infinity Fabric. The bandwidth per GPU is critical, but so is the interconnect bandwidth between GPUs for tensor parallelism.
Don't compare HBM bandwidth in isolation. Consider the entire memory hierarchy: L1, L2, HBM, and inter-GPU links. For multi-GPU inference, NVLink bandwidth (900 GB/s on H100) can become the bottleneck.
Future Outlook: HBM4, CXL, and the End of the Memory Wall?
HBM4, expected in 2026-2027, will double per-stack bandwidth to 1.6 TB/s and increase capacity to 64 GB per stack. NVIDIA's next-generation GPUs (likely Rubin or similar) will use HBM4, pushing total bandwidth to 6-8 TB/s. But even that won't solve the memory wall; it just kicks the can down the road. Emerging technologies like Compute Express Link (CXL) memory pooling allow multiple GPUs to share a pool of slower but larger memory (e.g., 2 TB of DDR5 at 500 GB/s). This is useful for models that don't fit in HBM, but the bandwidth gap between HBM and CXL is 10x, so performance will suffer. Another approach is near-memory computing, where compute logic is integrated into the HBM base die, reducing data movement. Samsung's HBM-PIM (Processing-in-Memory) and UCIe (Universal Chiplet Interconnect Express) are early steps. For now, the best strategy for AI builders is to optimize for HBM bandwidth: use quantization, efficient kernels, and batch serving to maximize tokens per second per dollar.
CXL memory pooling is useful for serving multiple models simultaneously, but don't expect it to match HBM performance for single-model inference.
Pitfalls and common misconceptions
- 1Misconception: More TFLOPS always means faster inference. Reality: For batch size 1, memory bandwidth dominates; compute only matters at high batch sizes.
- 2Misconception: HBM3E bandwidth doubles performance over HBM3. Reality: 1.6x bandwidth gives 1.6x tokens/sec only if the workload is perfectly memory-bound; real gains are 30-50%.
- 3Misconception: You need the latest GPU to run large models. Reality: An RTX 4090 with 1 TB/s GDDR6X can run 7B-13B models at 20-30 tok/s with quantization; HBM3E is for 70B+ models.
- 4Misconception: All HBM3E implementations are equal. Reality: H200's HBM3E runs at 4.8 TB/s, but MI300X's HBM3 runs at 5.2 TB/s; software maturity makes the difference.
- 5Misconception: Bandwidth is the only spec that matters. Reality: Capacity determines if a model fits on one GPU; without enough capacity, multi-GPU overhead can negate bandwidth gains.