Q8_0

Q8_0 is an 8-bit quantization type in the GGUF/llama.cpp ecosystem. It stands for '8-bit, version 0' and uses a block size of 32, with each block storing a scale in FP16 and 32 weight values as 8-bit signed integers. This scheme yields an effective average bitrate of about 8.5 bits per weight due to the overhead of the per-block scale. Q8_0 is often considered the 'sweet spot' for quantization because it preserves over 99.9% of the original FP16 model's accuracy for most tasks, while reducing memory usage by nearly 50% compared to FP16. This makes it ideal for scenarios where model quality is paramount, and memory savings are still beneficial, such as running larger models on high-end consumer GPUs with 24GB VRAM. In llama.cpp, Q8_0 is the default quantization level for many users who want to deploy models with minimal degradation. Compared to Q4_K_M, Q8_0 uses about twice the memory but offers significantly better perplexity, especially on smaller models. On the hardware side, Q8_0 benefits from INT8 tensor cores when available, but even on CPUs, integer operations are efficient. The format is widely supported across all GGUF-based tools. It is also commonly used as a cache format for KV-cache quantization in some implementations. For anyone uncertain about which quantization to use, Q8_0 is a safe bet that rarely disappoints in terms of quality, though memory-constrained setups may need to drop to Q4 or Q5 variants. The '0' in the name indicates the schema version, but it is the only widely deployed version in llama.cpp.