QAT vs PTQ: When to Retrain for Quantized LLMs

Introduction: The Quantization Landscape in 2026

By mid-2026, running large language models locally is the norm for serious AI builders. Whether you are deploying a 7B model on a Strix Halo NPU for edge inference or a 405B model across eight H200 GPUs, quantization is the single most impactful optimization for reducing memory and latency. The two dominant approaches are post-training quantization (PTQ) and quantization-aware training (QAT). PTQ applies quantization to a pre-trained model with minimal calibration data, while QAT incorporates quantization simulation into the training loop. For years, the ML community assumed QAT was always superior, but the reality for LLMs is more nuanced. With modern algorithms like GPTQ, AWQ, and GGML, PTQ often matches QAT at 8-bit and 4-bit weight-only quantization, especially for large models. However, as we push to sub-4-bit, activation quantization, and custom hardware, QAT becomes critical. This article examines the trade-offs with real numbers, hardware considerations, and practical recommendations for your deployment pipeline.

Post-Training Quantization: The Default for LLMs

Post-training quantization is the simplest path to reduced precision. You take a pre-trained model in FP16 or BF16, apply a calibration dataset (typically 128-1024 samples), and compute scale factors and zero points for each layer. The most popular methods for LLMs are GPTQ (Frantar et al., 2023) and AWQ (Lin et al., 2024), both of which use Hessian-based or activation-aware weight scaling to minimize the per-layer quantization error. GGML (Gerganov et al., 2023) offers a family of k-quant and i-quant formats that are widely used in llama.cpp for CPU and GPU inference. In practice, PTQ at 4-bit weight-only quantization achieves near-lossless perplexity for models like Llama 3 70B and Mistral 7B. For example, a 4-bit GPTQ quantized Llama 3 70B shows less than 0.1 perplexity increase on WikiText-2 compared to FP16, while reducing VRAM from 140 GB to 35 GB. This makes PTQ the default for anyone running models on consumer hardware like RTX 4090 (24 GB) or RTX 5090 (32 GB). The downside is that PTQ does not account for the interaction between quantization and downstream tasks, and it struggles with activation quantization (needed for integer-only inference on NPUs) and extreme low-bit widths (2-3 bits).

# Example: Quantize a model with GPTQ using AutoGPTQ
# Requires calibration data (e.g., c4 or wikitext)
# Outputs a quantized model that can be loaded in vLLM or transformers

autogptq --model_dir meta-llama/Llama-3.1-70B \
         --quant_method gptq \
         --bits 4 \
         --group_size 128 \
         --dataset c4 \
         --output_dir ./llama70b-4bit-gptq

Quantization-Aware Training: When and Why

Quantization-aware training injects fake quantization nodes into the forward pass during training, simulating the effects of low-precision arithmetic. The model learns to compensate for quantization noise, resulting in higher accuracy at the target bit width. QAT is mandatory for sub-4-bit quantization (2-3 bits) and for models that require integer-only inference (e.g., on NPUs without FP support). For example, Apple's MLX framework uses QAT to deploy Llama 3 8B at 3-bit on the M3 Ultra, achieving 15 tokens per second with only 6 GB of memory. Similarly, Qualcomm's AI Engine on Strix Halo requires QAT for INT8 activation quantization to meet latency targets. The cost is significant: QAT requires the original training dataset (or a representative subset), additional GPU hours (often 10-20% of the original training time), and careful hyperparameter tuning. For a 70B model, full QAT might require 500-1000 A100-hours. However, for models under 30B, QAT is feasible on a single node with 8 GPUs. The key insight is that QAT recovers 2-5% of the accuracy lost by PTQ at 4-bit, and up to 10% at 3-bit. For production systems where every percent of accuracy matters (e.g., medical or legal reasoning), QAT is worth the investment.

Hardware Realities: H100, RTX 5090, Strix Halo, and M3 Ultra

The hardware target heavily influences the quantization strategy. On NVIDIA H100/H200 with Tensor Cores supporting FP8 and INT8, PTQ to FP8 is trivial and lossless for most LLMs. The H100's Transformer Engine can automatically handle FP8 quantization during inference, making QAT unnecessary. For RTX 5090 (Blackwell), the new FP4 tensor cores (4-bit floating point) open the door to PTQ at FP4, though early results show a 1-2% accuracy hit on reasoning tasks. On AMD MI300X, ROCm supports INT8 and FP8 via MIOpen, but PTQ quality is slightly worse than NVIDIA due to less mature calibration libraries. The real differentiator is custom hardware like Apple M3 Ultra (Metal/MNX) and AMD Strix Halo NPU (XDNA). These platforms require integer-only inference (INT8, INT4) and often lack native FP16 support. For M3 Ultra, MLX uses QAT to produce 4-bit and 3-bit models that run efficiently on the unified memory architecture, achieving 20 tokens per second for a 70B model at 4-bit. For Strix Halo, the NPU demands activation quantization to INT8, which PTQ handles poorly for large models. QAT with quantization-aware fine-tuning (QAFT) can reduce perplexity increase from 0.5 to 0.1. if you are deploying on NVIDIA GPUs with FP8 support, PTQ is sufficient. If you target NPUs, ASICs, or Apple Silicon, invest in QAT.

Quantization Granularity: Group Size, Symmetric vs Asymmetric, and Mixed Precision

Both PTQ and QAT are affected by quantization granularity. Weight-only quantization typically uses per-channel or per-group scaling. Group size (e.g., 128, 32) determines how many weights share a scale factor. Smaller groups increase accuracy but also increase the overhead of storing scales. For PTQ, group size 128 is standard for 4-bit, achieving near-lossless quality. For QAT, you can use group size 32 to recover additional accuracy, at the cost of 10-15% more memory for scale factors. Symmetric quantization (zero point = 0) is simpler and faster on hardware, but asymmetric quantization can capture skewed distributions better. For activation quantization, symmetric is usually preferred because activations after ReLU/GELU are non-negative. Mixed precision is a powerful technique where sensitive layers (e.g., attention projections) are kept at higher precision. QAT naturally supports mixed precision by assigning different bit widths to different layers based on sensitivity analysis. PTQ can also do this via layer-wise calibration, but QAT allows the model to adapt to the mixed precision schedule. For example, a QAT-trained 3-bit model with 4-bit attention can match a 4-bit uniform model in accuracy while saving 15% memory.

# Example: Mixed precision QAT with PyTorch and torch.ao.quantization
# Assume model is a transformer; we quantize only the MLP layers to 4-bit
# and keep attention at 8-bit.

import torch
import torch.ao.quantization as quant

model = load_llama_model()
# Configure quantization for specific layer types
quant_config = quant.get_default_qconfig('x86')
for name, module in model.named_modules():
    if 'mlp' in name:
        module.qconfig = quant.get_default_qconfig('fbgemm')  # 4-bit
    elif 'self_attn' in name:
        module.qconfig = quant.get_default_qconfig('x86')     # 8-bit
    else:
        module.qconfig = None  # skip

# Prepare for QAT
model_prepared = quant.prepare_qat(model, inplace=False)
# Train for a few epochs with calibration data
train_model(model_prepared, dataloader, epochs=3)
# Convert to quantized inference
model_quantized = quant.convert(model_prepared, inplace=False)

Calibration Data and Validation: A Practical Comparison

PTQ requires a calibration dataset to compute quantization parameters. For LLMs, the standard is 128-1024 samples from the training corpus (e.g., C4, WikiText-2, or the Pile). The choice of calibration data matters: using domain-specific data (e.g., code for CodeLlama) can improve downstream accuracy by 0.5-1% on code generation tasks. QAT, on the other hand, uses the full training dataset or a large subset (e.g., 10% of the original). This gives QAT a fundamental advantage: it sees more diverse data and can adapt to long-tail distributions. In practice, QAT with 10% of the training data often matches PTQ with optimal calibration. However, obtaining the original training data is not always possible (e.g., for proprietary models). In that case, synthetic data generation using the model itself can help, but it introduces bias. For validation, perplexity on held-out text is the standard metric, but it does not capture task-specific performance. For reasoning tasks (e.g., GSM8K, MMLU), QAT typically shows a 1-3% improvement over PTQ at 4-bit. At 3-bit, the gap widens to 5-10%. If you care about benchmark scores, QAT is the safer bet.

Implementation Complexity and Tooling

PTQ is simple to implement with existing libraries. AutoGPTQ, ExLlamaV2, and llama.cpp provide one-command quantization for hundreds of models. AWQ is integrated into vLLM and TensorRT-LLM, allowing on-the-fly quantization during model loading. For NVIDIA GPUs, TensorRT-LLM offers INT4 and FP8 quantization with calibration, achieving up to 2x throughput over FP16. For AMD, ROCm's MIGraphX supports PTQ but with fewer options. QAT, in contrast, requires modifying the training loop. PyTorch's torch.ao.quantization provides QAT support for custom models, but for LLMs, the recommended approach is to use NVIDIA's TensorRT Model Optimizer (formerly TensorRT QAT) or the Hugging Face Optimum library with quantization trainers. For large models, QAT is typically done in two stages: first, PTQ to 8-bit, then QAT fine-tuning to 4-bit. This hybrid approach reduces training time by 50% compared to full QAT from scratch. The bottom line: if your team has ML engineering resources and access to GPUs, QAT is feasible. For solo developers or small teams, PTQ is the pragmatic choice.

Real-World Case Studies: 70B on Consumer GPU vs 405B on Datacenter

Consider two scenarios. First, a developer wants to run Llama 3 70B on a single RTX 5090 (32 GB VRAM). PTQ with 4-bit GPTQ reduces the model to 35 GB, but that still exceeds 32 GB. Using 3-bit GPTQ (26 GB) fits, but perplexity increases by 0.3-0.5. QAT at 3-bit can reduce the perplexity increase to 0.1, making the model viable for production. The developer spends 100 A100-hours on QAT (approx $200 on cloud), which is acceptable for a one-time cost. Second, a team deploys a 405B model on eight H200 GPUs (141 GB each) for a chatbot. With FP8 Tensor Cores, PTQ is lossless and achieves 50 tokens per second. QAT would require 2000+ A100-hours and offers negligible improvement. The team skips QAT. These cases illustrate the decision rule: use QAT only when you are forced to use very low bit widths (3-bit or lower) or when deploying on hardware without FP support.

Future Directions: FP4, NF4, and Hardware-Specific Quantization

The quantization landscape is evolving rapidly. NVIDIA's Blackwell (RTX 5090) introduces FP4 tensor cores, which support 4-bit floating point with a shared exponent. Early results show that FP4 PTQ matches INT4 accuracy for most models, with the advantage of simpler calibration (no zero point). However, FP4 is not supported on older hardware, so QAT for INT4 remains relevant. The NF4 format (normal float 4) used in QLoRA and bitsandbytes is a non-uniform 4-bit format that works well with PTQ for weight-only quantization. QAT for NF4 is possible but rarely used because NF4 is already near-lossless. On the hardware side, AMD's MI400 series is expected to support FP4, and Intel's Gaudi 3 supports INT4. For NPUs like Strix Halo, the trend is toward hardware-aware QAT where the quantization scheme is co-designed with the instruction set. Tools like MLX and MLC-LLM are leading this effort, providing automated QAT pipelines that target specific hardware. In the long term, we may see end-to-end learned quantization where the model learns its own optimal bit widths and scales, blurring the line between PTQ and QAT.

Practical Recommendations for Your Pipeline

Based on the above analysis, here is a decision framework. First, always start with PTQ. Use 8-bit weight-only quantization for maximum quality with minimal effort. If you need more memory savings, try 4-bit GPTQ or AWQ. Evaluate perplexity and task accuracy. If the degradation is acceptable (less than 1% on your benchmark), stop. If not, consider QAT. For models under 30B parameters and when deploying on NPUs or Apple Silicon, QAT is worth the investment. For models over 70B, QAT is only justified if you must go below 4-bit. Use hybrid approaches: start with PTQ to 8-bit, then QAT fine-tune to 4-bit. Leverage libraries like TensorRT Model Optimizer for NVIDIA, Optimum for Hugging Face, and MLX for Apple. Always validate on your target hardware with realistic workloads (e.g., long context, batch size 1). Finally, monitor the open-source ecosystem: as of 2026, llama.cpp supports QAT models via GGUF format, and vLLM can load QAT quantized models with custom kernels. The tooling is maturing fast, making QAT more accessible than ever.

Further reading

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• TensorRT Model Optimizer Documentation
• MLX: Array framework for machine learning on Apple Silicon