INT8

INT8 quantization converts 32-bit or 16-bit floating-point model weights and activations into 8-bit integer values, represented as signed or unsigned bytes. This reduces memory usage by 4x compared to FP32 and 2x compared to FP16, and on modern GPUs with INT8 tensor cores (like NVIDIA Turing and later architectures), it can double or quadruple inference throughput. For LLMs, INT8 quantization typically uses a per-tensor or per-row symmetric or asymmetric scaling scheme. The most common method is absmax quantization, where the absolute maximum value in a tensor is used to compute a scaling factor, then the FP values are multiplied by the scale and rounded to integers. A practical challenge is that LLM activations often have outliers, which can distort the quantization range-this is addressed by techniques like LLM.int8() which uses mixed-precision decomposition to handle outlier channels in FP16 while quantizing the rest. Another popular method is SmoothQuant, which smooths activation outliers to improve INT8 quality. On hardware, INT8 is highly optimized; for example, NVIDIA's Turing GPUs have INT8 tensor cores that can achieve up to 2x the throughput of FP16. Intel's Xe Matrix Extensions (XMX) also accelerate INT8. In the LLM ecosystem, INT8 is commonly used via libraries like llama.cpp (with Q8_0), TensorRT-LLM, and Hugging Face. The trade-off is accuracy: while many models retain near-lossless performance after INT8 quantization, some sensitive tasks (like code generation or reasoning) may degrade. Therefore, calibration with representative datasets is crucial. INT8 is especially beneficial for deploying models on edge devices and for cloud inference where reducing memory bandwidth is critical. For large models, fitting a 70B model into memory requires quantization below 8 bits, so INT8 is often a stepping stone rather than the final solution.