Attention

Attention is the core building block of transformer models, introduced in the 2017 paper 'Attention Is All You Need'. It enables each token to 'look at' all other tokens in the sequence, assigning relevance scores (weights) that determine how much each token influences the representation of the current one. The most common form is scaled dot-product attention: each token is projected into query (Q), key (K), and value (V) vectors; the dot product between a query and all keys gives attention scores, which are normalized via softmax and used to blend the values. This mechanism gives transformers their global receptive field-every output token can directly attend to any input token, unlike RNNs which process sequentially. Multi-head attention runs several attention heads in parallel, each learning different relationships (e.g., syntax, semantics, coreference). Attention's O(n^2) complexity in sequence length n is both its strength and weakness: it enables long-range dependencies but makes long-context inference expensive. Variants like causal attention (masked to prevent future token leakage) are used in autoregressive LLMs. FlashAttention reimplements the algorithm to reduce memory reads/writes, achieving up to 10x speedup for long sequences. Sparse attention patterns (e.g., sliding window, global+local) sacrifice some expressivity for efficiency. Attention is often visualized as heatmaps showing token-to-token relevance, useful for interpretability. In decoder-only models like GPT, each token's attention is computed over all preceding tokens in the context window.