LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large neural networks that avoids updating the full weight matrices of a pre-trained model. Instead, each target weight matrix W is held frozen while two small trainable matrices A and B are added alongside it, where the product AB has the same shape as W but is constrained to a low rank r. During the forward pass the effective weight becomes W + AB, and only the parameters of A and B are updated by gradient descent. Because r is typically 4 to 64 versus weight matrix dimensions of thousands, the number of trainable parameters shrinks by orders of magnitude.

Think of it like renovating a large building by inserting a handful of adjustable shims rather than replacing every beam and wall. The structure stays intact, but the small additions change how the whole thing behaves in precisely the ways you need.

A concrete example: fine-tuning a 7-billion-parameter LLM with full fine-tuning requires roughly 112 GB of GPU memory for mixed-precision weights plus optimiser states. With LoRA at rank 16 applied to the attention projection matrices only, the trainable parameter count drops to around 4 to 8 million, and the memory requirement falls to a level manageable on a single 24 GB GPU:

from peft import get_peft_model, LoraConfig
config = LoraConfig(r=16, lora_alpha=32,
                    target_modules=["q_proj", "v_proj"],
                    lora_dropout=0.05)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 4,194,304 / 6,738,415,616 (0.06%)

LoRA is a specialised form of Fine-tuning and sits within a family of parameter-efficient methods. It is closely related to Foundation Model adaptation, and its adapters can be merged back into the original weights at inference time with no added latency. Multiple LoRA adapters trained for different tasks can be hot-swapped or combined, a property exploited in Mixture of Experts-style serving setups. Understanding Gradient Descent and Overfitting helps in choosing appropriate rank and dropout values.

You will encounter LoRA when customising an open-source model for a specific domain, style, or task without access to the compute budget required for full fine-tuning.