Tokens per Second (TPS)

Tokens per second (TPS) measures the speed of an LLM during inference-how many output tokens are produced per second for text generation, or input tokens processed. It's a critical performance metric for user experience: higher TPS reduces perceived latency. TPS depends on hardware (GPU compute, memory bandwidth), model size (parameters), quantization, batch size, and sequence length. On a single A100 80GB with FP16 weights, a 7B parameter model might achieve ~80-100 TPS for generation, while a 70B model might deliver ~10-15 TPS. Quantizing to int4 or int8 can double or triple TPS by reducing memory bandwidth pressure. Techniques like FlashAttention and KV cache optimizations also improve TPS, especially for long sequences. For batched inference, TPS is often reported as total tokens across all requests per second (aggregate throughput) or per-user tokens (throughput / batch size). There's a fundamental tradeoff with latency: maximizing TPS via large batches increases per-request latency, which is unacceptable for real-time apps. AI builders must balance user-facing latency (e.g., <200ms per token) with backend throughput (e.g., hundreds of TPS). Profiling TPS across different batch sizes, sequence lengths, and hardware is essential for capacity planning. Tools like vLLM, TensorRT-LLM, and llama.cpp report TPS. Remember that output tokens cost more than input because of autoregressive generation.