LoRA (Low-Rank Adaptation)

LoRA (Low-Rank Adaptation) revolutionizes how large language models are customized. Instead of updating all model parameters during fine-tuning — which is prohibitively expensive for billion-parameter models — LoRA adds a small number of trainable parameters while freezing the original weights.

Key advantages:

• Efficiency: Fine-tune a 70B parameter model on a single GPU
• Size: LoRA adapters are typically 1-10 MB vs gigabytes for full model weights
• Portability: LoRA weights can be swapped in and out like add-on modules
• No Catastrophic Forgetting: The original model knowledge is preserved

LoRA belongs to the family of Parameter-Efficient Fine-Tuning (PEFT) methods alongside Prefix Tuning, Adapters, and QLoRA (LoRA combined with quantization). LoRA has become the standard approach for open-source model customization, powering thousands of fine-tuned variants on platforms like Hugging Face.