Before you start
- ✓Basic familiarity with what an LLM is
- ✓No math required
The one-line summary
Quantization is the practice of storing model weights in fewer bits than the model was trained at. The trade is: fewer bits = smaller file, less VRAM, faster inference, but lower quality. The sweet spot for local hardware is 4-bit quantization (specifically Q4_K_M in GGUF format), which uses about a quarter of the memory of full precision while keeping 97-99% of the model's quality on typical benchmarks. Below 4 bits you start seeing real quality loss; above 4 bits the returns diminish quickly.
Why models are trained in FP16 (or BF16)
When a model is trained, each parameter is a 16-bit floating point number. Llama 3.1 70B in BF16 is exactly 140 GB on disk (70 billion params × 2 bytes per param). FP16/BF16 is the lowest precision that's stable for training, go lower and gradients explode. Inference is more forgiving than training, so we can use fewer bits at runtime without retraining the model. That's what quantization exploits.
FP16 (half precision) and BF16 (brain float 16) are both 16-bit but distribute the exponent/mantissa differently. For inference it doesn't matter, they perform identically. BF16 just has a wider range, useful for training.
How Q4 actually works
A 4-bit quantization scheme replaces each FP16 weight with one of 16 possible values (since 2^4 = 16). You can't just lop off bits, you have to find a way to represent a range of FP16 weights with those 16 slots. The simplest scheme picks a min and max and divides the range into 16 buckets. Modern schemes like Q4_K_M are smarter: they group weights into blocks of 32 or 64, find the optimal scale per block, and use mixed bit-widths for different layers (more bits where it matters, fewer where it doesn't).
GGUF quant table (the practical reference)
GGUF is the file format used by llama.cpp and Ollama. Within GGUF, quants are named by a convention: Q[bits]_K_[size]. K stands for the K-quant family (the modern smart quants). S/M/L is the block size, Large blocks store more per block, less metadata overhead, but slightly lower quality. Practical table for a 70B model:
Quant Size Quality(%) VRAM (70B) Use when
----------------------------------------------------------------------
FP16 140 GB 100 140 GB Never, on consumer HW
Q8_0 70 GB 99.9 70 GB 2× 48GB cards, peak quality
Q6_K 58 GB 99.7 58 GB 2× 32GB or 1× 48GB pro card
Q5_K_M 48 GB 99.0 48 GB 2× 24GB consumer (3090/4090 pair)
Q4_K_M 40 GB 97.5 40 GB Best balance, DEFAULT
Q4_K_S 38 GB 96.0 38 GB Tight on memory, OK quality
Q3_K_M 32 GB 92.0 32 GB Last-resort fit, real degradation
Q2_K 26 GB 80.0 26 GB Don't.Quality is approximate, measured on perplexity for Llama 70B against FP16 baseline.
Q4_K_M is the default and that's correct
For 99% of users running 99% of models, Q4_K_M is the right choice. It fits in standard consumer VRAM budgets, the quality loss vs the FP16 model is barely measurable in chat/coding tasks, and it's the best-tested quant by far (every model on Hugging Face that releases GGUF builds includes Q4_K_M). You should only deviate if you have a specific reason: a research workload that needs every basis point of quality (go Q6 or higher), or a fit problem where Q4 doesn't quite fit (go Q3 reluctantly).
When 4-bit hurts: long context
Quantization affects weights, but inference also needs a KV cache (the model's working memory of the current conversation). For long contexts the KV cache grows linearly and can become as big as the model weights. The KV cache is usually FP16, quantizing it too gives you back a lot of memory. Set OLLAMA_KV_CACHE_TYPE=q8_0 to use 8-bit KV cache, which roughly halves its size with almost no quality loss. q4_0 KV cache is more aggressive, useful for very long contexts but starts to show in long conversations.
AWQ, GPTQ, and EXL2, the other quant families
GGUF/llama.cpp is one ecosystem. The other big ones are AWQ (Activation-aware Weight Quantization, used by vLLM and Transformers) and GPTQ (old school, mostly replaced). AWQ is roughly the same quality as Q4_K_M GGUF and is what you want if you're using vLLM. EXL2 is ExLlamaV2's format, extremely fast on NVIDIA, supports unusual bit widths like 4.65bpw. For Ollama users you don't need to think about these; for vLLM users grab the AWQ build of your model when available.
The smaller the model, the bigger the hit from quantization
Counterintuitive but important: Q4 hurts a 7B model proportionally more than it hurts a 70B model. 70B has redundant capacity that absorbs quantization noise; 7B is already compressing its knowledge tight and has no slack. So you'll see communities still using Q8 or even FP16 for tiny models (under 3B) while running Q4 for everything 7B+. As a rule: 1B → Q8, 3B → Q6, 7B → Q5 or Q4_K_M, 13B+ → Q4_K_M is fine.
Quality testing your specific use case beats any table. If chat feels off, try a higher quant and see if the feeling goes away.
Tags
Stuck? Share your build?
Hundreds of homelabbers are working through these same tutorials in our community. Drop your config, ask the hard question, or show off what you built.
Join the discussion
