Transformer

The Transformer architecture, introduced in the 2017 paper 'Attention Is All You Need', revolutionized natural language processing by replacing recurrent neural networks (RNNs) with self-attention. Its key innovation is the use of multi-head self-attention and feed-forward layers stacked in encoder-decoder blocks. Self-attention computes the relevance of every token in a sequence to every other token, allowing the model to capture long-range dependencies without sequential recurrence. This enables parallel training on GPUs, drastically reducing training time. The Transformer consists of an encoder (which processes the input) and a decoder (which generates output), but many modern LLMs (like GPT) use only the decoder stack, while BERT uses only the encoder. Each layer includes layer normalization and residual connections to stabilize training. The key parameters: number of layers (depth), hidden dimension (width), number of attention heads, and feed-forward size. Scaling laws show that larger models (more parameters) with more data yield better performance, but inference cost grows quadratically with sequence length due to attention's O(n) complexity. Efficient variants like sparse attention, FlashAttention, and linear attention tackle this. The Transformer's versatility extends to vision (ViT), audio (Whisper), and multimodal models. For AI builders, understanding Transformer internals helps optimize batch size, sequence length, and hardware choice (e.g., HBM bandwidth for attention operations).