FlashAttention

FlashAttention is a groundbreaking algorithmic reformulation of the standard attention computation, designed to exploit the memory hierarchy of modern GPUs. Standard attention materializes the full N x N attention matrix in HBM (high-bandwidth memory), which is both memory-prohibitive for long sequences and bandwidth-bound. FlashAttention works by tiling the Q, K, and V matrices across GPU on-chip SRAM (shared memory), computing attention in small blocks, and recombining results without ever writing the full softmax matrix to HBM. It uses two key tricks: (1) recomputation of attention during backward pass to avoid storing intermediate values (a form of gradient checkpointing), and (2) on-the-fly softmax normalization using a running rescaling technique. The result is a 2-10x wall-clock speedup for attention, and memory savings that make it practical to handle context windows of 128K+ tokens on a single A100. Two variants exist: FlashAttention-1 (forward only) and FlashAttention-2 (improved backward pass, better threadblock scheduling). FlashAttention-3 adds FP8 support and asynchronous processing on Hopper GPUs. For AI builders, integrating FlashAttention into training or inference pipelines is nearly drop-in via libraries like xformers, PyTorch's scaled_dot_product_attention, or Hugging Face's Transformers. It is especially beneficial for long-document RAG, code generation with large contexts, and multi-turn chat with extensive history. Note that FlashAttention requires Ampere or newer GPUs (compute capability 8.0+), and may show small numerical differences due to block-wise accumulation order.