LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning technique that adds small, trainable matrices to existing model weights, enabling fine-tuning of large models with minimal computational cost.
LoRA (Low-Rank Adaptation) revolutionizes how large language models are customized. Instead of updating all model parameters during fine-tuning — which is prohibitively expensive for billion-parameter models — LoRA adds a small number of trainable parameters while freezing the original weights.
Key advantages:
- Efficiency: Fine-tune a 70B parameter model on a single GPU
- Size: LoRA adapters are typically 1-10 MB vs gigabytes for full model weights
- Portability: LoRA weights can be swapped in and out like add-on modules
- No Catastrophic Forgetting: The original model knowledge is preserved
LoRA belongs to the family of Parameter-Efficient Fine-Tuning (PEFT) methods alongside Prefix Tuning, Adapters, and QLoRA (LoRA combined with quantization). LoRA has become the standard approach for open-source model customization, powering thousands of fine-tuned variants on platforms like Hugging Face.