AWQ

AWQ (Activation-aware Weight Quantization) was developed by MIT and NVIDIA and published in 2023. Unlike GPTQ, which treats all weights equally, AWQ observes that a small fraction of weight channels are significantly more important than others based on the magnitude of their activations. AWQ identifies these salient weight channels by analyzing a calibration dataset's activation patterns, and then scales these important channels up before quantization, reducing the relative quantization error. This scaling is compensated by dividing the downstream layers' weights accordingly, so the output remains mathematically equivalent. The method is per-channel, meaning it scales entire channels (not individual weights), which makes it efficient to implement. AWQ achieves better perplexity than GPTQ at the same bitrate, especially for 4-bit and 3-bit quantization. It also works natively with group-wise quantization, using group sizes like 128. AWQ does not require any training; it is a post-training optimization using a small calibration set (e.g., 128 sequences from the Pile). The quantized models are often stored in the same format as GPTQ (using the AutoGPTQ loader) or in a dedicated AWQ format. Inference speed is comparable to GPTQ, with some implementations showing minor throughput advantages due to simpler kernel requirements. AWQ is supported in vLLM, TensorRT-LLM, ExLlama (v2), and Hugging Face. It is particularly popular for 4-bit quantization of instruction-tuned models, where it preserves quality remarkably well. For AI builders, AWQ is often the preferred method for deploying quantized models with the highest quality at 4 bits, especially when the model is sensitive to quantization errors, such as code or math models. The main downside is slightly longer calibration time compared to simpler methods, but it is still a matter of minutes.