High Bandwidth Memory (HBM)

High Bandwidth Memory (HBM) is a type of 3D-stacked DRAM (dynamic random-access memory) used in modern GPUs (H100, A100, MI300) to deliver extremely high data transfer rates-up to 3.35 TB/s on H100 HBM3, compared to ~50 GB/s for conventional GDDR6. This bandwidth is critical for LLM inference because transformer models are often memory-bound: each token generated requires reading the full model weights from HBM into compute units. HBM's stacked design (up to 12 dies) reduces physical footprint and power consumption while providing wide bus interfaces (1024-bit). For training, HBM enables large batch sizes and model parallelism without stalling compute. However, HBM capacity is limited (e.g., 80GB on A100, 94GB on H100). To fit large models (e.g., 70B parameters in FP16 ~140GB) you need multiple GPUs and model parallelism, which also requires fast interconnects like NVLink to share HBM pools. Quantized models (int4, int8) reduce HBM pressure-a 70B in int4 uses ~35GB. HBM's bandwidth determines tokens-per-second: roughly, throughput = bandwidth / (weights per token). For AI builders, optimizing for HBM bandwidth means using tensor cores (FP8/INT8), minimizing memory reads via kernel fusion, and using optimized attention (FlashAttention). HBM also has higher latency than SRAM (on-chip cache) but much lower than system RAM. In multi-GPU setups, HBM bandwidth per GPU directly scales overall throughput linearly.