vLLM

vLLM is an open-source high-throughput and memory-efficient inference engine for large language models, developed at UC Berkeley. It is designed to serve LLMs in production environments, offering features that significantly improve performance over naive implementations. At its core, vLLM introduces PagedAttention, a novel attention algorithm that manages the KV cache in fixed-size blocks (pages), similar to virtual memory in operating systems. This eliminates memory fragmentation and allows near-100% utilization of GPU memory for the KV cache, which is typically the bottleneck in LLM serving. For AI builders, vLLM supports most popular transformer-based LLMs including Llama, Mistral, Mixtral, GPT-NeoX, and Falcon. It integrates smoothly with Hugging Face Transformers and can be used via its own Python API or through OpenAI-compatible endpoints. vLLM supports continuous batching, where incoming requests are queued and processed together to maximize GPU utilization, and it can dynamically allocate KV cache blocks across requests. It also supports tensor parallelism and pipeline parallelism for multi-GPU deployments, as well as various quantization methods like GPTQ, AWQ, and FP8. Performance-wise, vLLM can achieve up to 24x higher throughput than Hugging Face Transformers for serving LLMs. It also includes prefix caching to speed up repeated prompts and supports speculative decoding. The engine is optimized for both latency-sensitive and throughput-heavy applications. vLLM's popularity stems from its production readiness, extensive documentation, and active community. It is used by companies like Meta, Salesforce, and Replicate.