GPTQ

GPTQ (Generative Pre-trained Transformer Quantization) is a one-shot weight quantization method for large language models, introduced by Frantar et al. in 2023. It uses an Optimal Brain Quantization (OBQ) framework, which greedily quantizes weights while updating remaining weights to compensate for the introduced error. The result is remarkably accurate 3-bit and 4-bit quantized models that often match FP16 quality. GPTQ is performed after training, requiring a small calibration dataset (typically 128 samples) to compute quantization parameters. The quantization process is layer-wise: for each linear layer, GPTQ selects the order in which weights are quantized to minimize the overall loss in output activations. GPTQ supports group size (e.g., 128) as a hyperparameter; smaller groups improve accuracy but increase memory overhead for scale and zero-point storage. The output of GPTQ is a compressed model, commonly stored in the GPTQ format (separate from GGUF), often with files like 'model-4bit-128g.safetensors'. GPTQ models are supported by vLLM, ExLlama, Hugging Face Transformers, and AutoGPTQ. Inference speed is excellent on GPU, but GPTQ models generally require GPU for practical inference; they do not run as efficiently on CPU compared to GGUF. The main trade-off is that GPTQ's quality degrades more rapidly below 4 bits, so 3-bit and 2-bit versions are rarely used. GPTQ is popular for deploying quantized models on dedicated GPU servers where maximum throughput is needed, and memory is at a premium. It is also the basis for many commercial inference APIs. The method has been extended to handle activation quantization as well, though pure weight-only GPTQ remains dominant. For AI builders, GPTQ is a mature and well-supported quantization path, especially for batch inference with high concurrency.