LoRA (Low-Rank Adaptation)

LoRA (Low-Rank Adaptation) is a technique for efficiently adapting large pre-trained models to new tasks without modifying the original weights. Instead of updating the full parameter matrix (e.g., 4096x4096 in a dense layer), LoRA injects a pair of smaller matrices A and B such that their product (an n x r and r x m matrix) is added to the original weights. The rank r (typically 8-128) is much smaller than the original dimensions, drastically reducing trainable parameters by 100-1000x. During training, only these low-rank matrices are updated, while the original model remains frozen. This enables fine-tuning on consumer hardware-for example, fine-tuning a 7B model on a single RTX 4090 with LoRA, compared to needing 8x A100 for full fine-tuning. LoRA is typically applied to the query and value projection matrices in attention, but can target any linear layer. Multiple LoRA adapters can be hot-swapped during inference, enabling rapid task switching (e.g., a coding adapter, a medical adapter, a creative writing adapter). Variants include QLoRA (quantization + LoRA), DoRA (weight-decomposed), and AdaLoRA (adaptive rank allocation). LoRA adapters are tiny (often under 100MB) and easily distributable via Hugging Face. The primary trade-off: aggressive rank reduction can limit expressivity for tasks far from the original training distribution. AI builders often rank-search (trying r=32, 64, 128) and use target_modules='all-linear' for best performance. LoRA is also used in RLHF pipelines to efficiently reward model train.