Q4_K_M

Q4_K_M stands for '4-bit, K-quant, Medium', part of a family of K-quant quantization types in llama.cpp designed by Gerganov and team. It divides weights into blocks of 32, and within each block, it uses a super-block structure: a 6-bit scale (super-block), 4-bit sub-blocks, and a 4-bit minimum value for the super-block. The method is 'mixed' in that some very important weights are quantized to 6-bit precision to preserve quality, while the majority stay at 4 bits. Specifically, each super-block contains 32 weights, stored as 4-bit integers plus a 6-bit scale and 4-bit min. This achieves an effective bitrate around 4.5 bits per weight, but with superior quality compared to naive 4-bit quantization. Q4_K_M is the recommended 4-bit quantization level in llama.cpp because it provides a balanced trade-off between model size, inference speed, and perplexity. In benchmarks, Q4_K_M often matches or exceeds the quality of other 4-bit methods like Q4_0, while only using slightly more memory. The 'K' stands for K-quant, which is an umbrella term for quantization schemes that use a combination of per-block scales and min values to compress groups of weights. Compared to Q4_K_S (smaller, lower quality) and Q4_K_L (larger, higher quality), Q4_K_M sits in the middle and is the default for many users. It is widely used to run models like Llama 3, Mistral, and CodeLlama on consumer hardware. Inference speed is excellent, often achieving 20-40 tokens per second on a mid-range GPU. The format is natively supported in llama.cpp, Ollama, LM Studio, and other GGUF-based tools. When choosing a quantization level, Q4_K_M is often the first recommendation for local LLM deployment.