Context Window

The context window is the fixed-size sequence length that a transformer model was trained to handle, typically 2048 to 1 million tokens in modern LLMs. It limits how many tokens - from prompt, conversation history, or retrieved documents - can be attended to simultaneously. When input exceeds this length, the model must truncate, chunk, or use sliding window strategies. The context window size is limited by the quadratic memory complexity of full attention (O(n^2) with n tokens) and by positional encoding schemes like RoPE, which degrade beyond a certain length. Long-context models (e.g., GPT-4 128K, Claude 200K, Gemini 1M) achieve their reach via techniques like ALiBi, YaRN, or RingAttention to extrapolate to longer sequences. For RAG pipelines, context window size directly impacts how many retrieved document chunks can be inserted into the prompt; exceeding it forces summarization or chunk dropping. In inference, a larger context window dramatically increases KV cache memory usage, often by gigabytes per request. AI builders must balance context window size with available VRAM and throughput requirements. Fine-tuning with RoPE scale factors or using models with linear attention can extend windows without retraining, but may degrade performance on shorter sequences. In practice it refers to the combined length of prompt and generated tokens - exceed it and the oldest tokens are dropped, breaking long conversations or large code-review tasks.