INT4

INT4 quantization reduces each weight to 4 bits, representing only 16 possible values per weight. This aggressive compression allows massive models-like 70B parameter LLMs-to fit into GPUs with 24GB VRAM or even run on CPUs. The key challenge is maintaining model quality when using only 16 discrete levels per weight. Modern methods (like GPTQ, AWQ, and GGUF's Q4_K_M) address this through group-wise quantization, where weights are divided into small groups (e.g., 32 or 128 weights) and each group has its own scaling factor and zero-point. This local adaptivity preserves precision where it matters. INT4 models typically use asymmetric quantization, storing both scale and zero-point in FP16, which adds a small overhead but greatly improves accuracy. For instance, Q4_K_M in llama.cpp uses a 4-bit block quantization with a median of 4-bit values and some higher precision for important weights. Another approach is NF4 (NormalFloat4), used in QLoRA, which is information-theoretically optimal for normally distributed weights. In practice, INT4 models achieve token-per-second throughput that is 2-3x higher than FP16 on memory-bandwidth-bound systems, since less data needs to be loaded from VRAM per token. However, compute-bound scenarios (e.g., very large batch sizes) see less benefit. Accuracy loss from INT4 is model-dependent; many instruction-tuned models show minimal degradation, while base models may lose some perplexity. Techniques like weight-only quantization (where only weights are quantized but activations remain FP16) are common to reduce memory pressure without increasing compute complexity. INT4 is the go-to choice for local deployment (e.g., on a consumer RTX 3090) because it balances memory savings and quality. It is also used in cloud inference to increase server throughput by fitting more models on a single GPU.