Throughput vs Latency

Throughput and latency are opposing forces in LLM inference. Throughput (e.g., requests per second or tokens per second) gauges system capacity-how much work is done in a given time. Latency (e.g., time-to-first-token, TTFT, and inter-token latency) reflects user experience from a single request. Increasing batch size improves throughput by utilizing GPU arithmetic units fully, but it also raises latency because each request waits longer for the batch to fill and for matrix computations to complete. For real-time applications (chatbots, copilots), latency targets are tight: TTFT under 500ms, per-token generation under 30ms. Offline batch processing (data labeling, summarization) prioritizes throughput over latency. Techniques to navigate this: dynamic batching (combine requests arriving close in time), continuous batching (vLLM), and kernel optimizations like FlashAttention reduce latency without sacrificing throughput. Quantization lowers memory bandwidth use, helping both metrics. Speculative decoding trades throughput for perceived latency by generating draft tokens faster. AI builders must define Service Level Objectives (SLOs) like p50 TTFT and p99 total latency, then tune batch size, hardware count, and model size to meet them. Use tools like NVIDIA's FasterTransformer or TensorRT-LLM for profiling. A common pitfall: optimizing for throughput alone degrades user experience, while chasing zero latency at single-batch wastes compute resources.