GGUF

GGUF (GPT-Generated Unified Format) is a file format for storing large language models, created by the llama.cpp team to replace the older GGML format. It was introduced in August 2023 to address limitations in GGML, such as the inability to add new quantization types without breaking backward compatibility. GGUF is essentially a container format that stores the model's tensor weights, tokenizer, hyperparameters, and metadata in a single file. The format is extensible, allowing tool developers to define custom metadata keys. Key design features include: versioning (currently up to GGUF v3), support for many quantization types (Q4_0, Q4_K_M, Q5_K_M, Q8_0, etc.), and efficient loading with memory-mapped I/O. This last feature is critical-GGUF files can be loaded by mapping the file into virtual memory, enabling instant model loading and sharing of memory pages across processes. The format also supports parallelization-friendly tensor layouts. GGUF has become the de facto standard for local LLM deployment on CPU and Apple Silicon, used by Ollama, LM Studio, llama.cpp, and many other tools. Compared to Hugging Face's safetensors format, GGUF is more focused on inference optimization rather than training. One major advantage is that GGUF files are self-contained: you don't need separate configuration files or tokenizer files. For AI builders, using GGUF means you can download a single file and run an LLM immediately. The quantization metadata within GGUF also allows inference engines to automatically select the right kernels. The format is maintained by the llama.cpp community and is open-source.