llama.cpp

llama.cpp is a high-performance, open-source implementation of LLaMA-based large language models written in C/C++. It was created by Georgi Gerganov and has become one of the most popular tools for running LLMs locally on consumer hardware, including laptops, desktops, and even Raspberry Pis. Its key innovation is the GGUF format (replacing the older GGML format), which packages model weights, tokenizer, and metadata into a single file with support for multiple quantization levels. llama.cpp supports a wide range of quantization types, including Q2_K, Q3_K, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and FP16. These allow models to be compressed from their original FP16 size (e.g., 7B model ~13GB) down to as little as 3-4GB while retaining reasonable quality. The engine supports CPU inference using SIMD optimizations (AVX2, AVX512, NEON) and GPU acceleration via backends for CUDA, Metal, Vulkan, ROCm, and SYCL. It also supports multi-GPU setups, speculative decoding, and batching for batched inference. For AI builders, llama.cpp is often used for prototyping, local development, and embedded applications where cloud GPUs are unavailable. It provides a simple command-line interface and an HTTP server with an OpenAI-compatible API. llama.cpp can also be compiled as a library and linked into other applications, making it a versatile building block. It supports various model architectures, including Llama 2/3, Mistral, Gemma, Phi, and many more. Performance is competitive with vLLM for single-user scenarios, though vLLM typically excels in high-throughput server settings.