ExLlama v2

ExLlama v2 (ExL2) is an advanced inference framework designed specifically for Llama-based large language models on NVIDIA GPUs. It combines a custom quantization format with a highly optimized CUDA kernel to achieve state-of-the-art performance in both speed and memory efficiency. The ExL2 format supports arbitrary bit-widths from 2 to 8 bits per weight, allowing fine-grained trade-offs between model size and perplexity. Unlike older formats like GPTQ or AWQ, ExL2 uses a block-wise quantization scheme with per-group scaling factors, enabling better preservation of outlier weights. Its key innovation lies in the inference engine: ExL2 implements a fused attention kernel that overlaps memory loads with computation, and it dynamically manages the KV cache across CPU and GPU memory to maximize throughput under limited VRAM. The engine also supports speculative decoding and batch inference natively. For AI builders, ExL2 is often the fastest choice for deploying 7B to 70B parameter models on consumer GPUs (e.g., RTX 3090/4090), achieving tokens-per-second rates up to 2x faster than similar quantized formats. It is particularly well-suited for real-time chat applications and local RAG pipelines where low latency and high throughput are critical. However, ExL2 quantized models may show slightly higher perplexity than the full-precision version, especially at lower bit depths. The format is tightly coupled with the Hugging Face ecosystem through the ExLlama library, and many quantized models are distributed as .safetensors in ExL2 format.