Batch Size

Batch size determines how many input sequences are processed in parallel by the model on a given GPU iteration. During training, larger batch sizes produce more stable gradients, enable higher learning rates, and better utilize parallel hardware (tensor cores). However, they demand more GPU memory for activations and gradients. Typical batch sizes range from 8 to 2048 depending on model size and hardware memory. For inference, batch size directly influences throughput vs latency tradeoff: larger batches increase tokens-per-second overall but increase per-request latency because all requests must be padded to the same sequence length (or use dynamic batching). Techniques like continuous batching (vLLM) allow each sequence to run independently, reducing wasted computation on padding tokens. For autoregressive decoding, batch size multiplies the KV cache memory footprint linearly-a batch of 32 with 4096 tokens per sequence demands 32x more cache than a single sequence. Thus, memory-bound models (e.g., 70B parameters) may fit only a small batch on one GPU. For AI builders, profiling optimal batch size is crucial: start with batch=1, increase until GPU memory is 90-95% utilized, then observe throughput saturation. Precision (FP16, int8) also changes effective batch size limits. Batch size also affects quantization calibration: you need representative batch samples for per-tensor or per-channel scaling.