Glossary/ Quantization

Q5_K_M

Q5_K_M is a 5-bit mixed-precision quantization method in llama.cpp that offers higher accuracy than 4-bit methods with moderate memory increase.

Q5_K_M is part of the K-quant family in llama.cpp, providing 5-bit quantization with a medium configuration. It uses a block size of 32, with each super-block containing a 6-bit scale, a 5-bit minimum value, and the majority of weights stored as 5-bit integers. Similar to Q4_K_M, some important weights may be stored at higher precision within the block structure, yielding an effective bitrate around 5.5 bits per weight. This makes Q5_K_M an excellent choice when you need higher fidelity than 4-bit quantization but cannot afford the full 8-bit or FP16 memory footprint. For example, a 70B parameter model in Q5_K_M requires approximately 48 GB of memory, compared to 39 GB for Q4_K_M and 140 GB for FP16. The perplexity improvement over Q4_K_M is often noticeable, especially on complex reasoning and code-generation tasks. In practice, Q5_K_M is popular for running models on MacBooks with unified memory (e.g., M2/M3 with 64-96 GB) or on dual-GPU setups. The inference speed is slightly slower than Q4_K_M due to the increased data transfer, but still substantially faster than FP16. Many open-source LLM enthusiasts use Q5_K_M as the 'safe' quantization that retains nearly all original model quality. The method is available in llama.cpp, Ollama, and LM Studio. When selecting quantization levels, if you have spare memory and want the best quality without moving to 8-bit, Q5_K_M is the recommended pick. It is also a common option for quantization benchmarks, often outperforming GPTQ 4-bit models in terms of accuracy.

Common misconception

Q5_K_M is exactly 5 bits per weight; it actually uses about 5.5 bits per weight due to block overhead and mixed precision.

Canonical reference

https://github.com/ggml-org/llama.cpp#quantization

Related terms

Concepts you'll usually encounter alongside this one.

Q4_K_M

Quantization

Q4_K_M is a 4-bit quantization method in llama.cpp that combines 4-bit quantization for most weights with higher precision for important weights using a block size of 32.

Q8_0

Quantization

Q8_0 is an 8-bit integer quantization format in llama.cpp that balances high accuracy with good compression, offering near-lossless quality for many LLMs.

GGUF

Model Format

GGUF is a binary format for storing quantized LLM models, designed for efficient CPU and GPU inference, superseding GGML.

INT4

Quantization

INT4 is a 4-bit integer quantization format that compresses model weights by 8x vs FP32, enabling large LLMs to run on consumer hardware with modest quality loss.