QLoRA (Quantized LoRA)

QLoRA (Quantized Low-Rank Adaptation) extends LoRA by first quantizing the frozen base model to 4-bit precision (using the NF4 data type), then applying standard LoRA adapters in full (FP16/BF16) precision. This dramatically reduces GPU memory requirements: fine-tuning a 65B LLaMA model requires only 48 GB, fitting on a single A100, whereas full fine-tuning would need 8x A100s. QLoRA introduces the NormalFloat4 (NF4) data type, a distribution-adaptive quantization scheme that better preserves information for normally distributed weights. It also employs double quantization (quantizing the quantization constants) to further reduce memory overhead, and paged optimizers (using unified memory) to handle gradient checkpointing spikes. The key insight: since LoRA adapters are low-rank and small (typically <1% of original parameters), they can be kept at high precision while the massive base model is heavily compressed. The resulting fine-tuned model quality is nearly indistinguishable from full fine-tuning on target tasks, with perplexity differences often under 0.5%. QLoRA was a breakthrough paper in 2023, enabling widespread consumer-grade fine-tuning of models like LLaMA-2 and CodeLlama. For AI builders, QLoRA is the go-to method for domain adaptation (e.g., legal, medical, code) without GPU clusters. However, training speed is slower than full-precision LoRA due to quantization/dequantization overhead, and batch sizes must be small. Fine-tuning for very long contexts (>4K tokens) may still require multiple GPUs due to KV cache growth.