LoRA, short for Low-Rank Adaptation, is a technique that makes fine-tuning large pre-trained models practical on modest hardware by decomposing weight updates into pairs of small matrices rather than modifying the original weights directly. Formally, for a weight matrix W of dimensions d by k, LoRA adds a bypass path consisting of a d-by-r down-projection matrix A and an r-by-k up-projection matrix B, where the rank r is much smaller than d or k. Only A and B are trained; W stays frozen. At inference the adapted weight is W + (alpha/r) * AB, and the adapter can be merged into W so there is zero additional latency in production.

Think of it like writing margin notes in a textbook. The printed text (the pre-trained weights) is unchanged, but your annotations (the low-rank matrices) guide how you interpret and respond to new questions — and you can share just the annotations rather than reprinting the whole book.

The practical impact is significant. Fine-tuning Meta's Llama 3 8B on a domain-specific dataset with LoRA requires roughly 16 GB of VRAM versus 150+ GB for full fine-tuning with AdamW optimiser states. Community model hubs like Hugging Face host thousands of LoRA adapters that can be downloaded and loaded on top of a base model in seconds:

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
model.load_adapter("username/my-lora-adapter")  # ~16 MB file

LoRA is a member of the parameter-efficient Fine-tuning family and shares goals with Knowledge Distillation — both reduce the resource cost of adapting large models. The choice of which layers to apply adapters to (typically the Self-Attention projection matrices) affects what kinds of behaviour change during adaptation. Rank selection is a Hyperparameter decision, and Overfitting is a risk when the training dataset is small relative to the rank chosen.

You will encounter LoRA adapters throughout the open-source AI ecosystem, from image generation models fine-tuned on specific art styles to chat models adapted for medical, legal, or coding domains.