Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques for adapting a pretrained model to a new task or domain by updating only a small fraction of its parameters rather than all of them. Full fine-tuning of a model with tens or hundreds of billions of parameters requires enormous compute and storage — one full copy of updated weights per task. PEFT methods instead introduce a small set of trainable parameters (adapters, low-rank matrices, soft prompt tokens, or selective layer unfreezes) while keeping the bulk of the original weights frozen, dramatically reducing memory use and training time without sacrificing much task performance.

Think of it like customizing a printed circuit board by soldering on a small daughter board rather than manufacturing an entirely new main board from scratch. The base board does most of the work unchanged; the daughter board adds the specialized behavior you need for one particular device.

The most widely used PEFT technique is LoRA (Low-Rank Adaptation), which injects trainable rank-decomposition matrices into the attention layers of a transformer. In practice:

from peft import get_peft_model, LoraConfig, TaskType
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"], task_type=TaskType.CAUSAL_LM)
peft_model = get_peft_model(base_model, config)
peft_model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,034,516,992 || trainable%: 0.052

Only 0.05% of parameters are updated, yet the model can be meaningfully adapted to a new style or domain. The resulting adapter weights are a few tens of megabytes rather than gigabytes.

PEFT is most closely related to LoRA, which is its dominant implementation, and to Fine-tuning broadly. It is also tightly coupled to Parameter count concerns and to Foundation Model reuse, since PEFT makes sense only when starting from a large pretrained base.

You will encounter PEFT whenever adapting large language or vision models to specialized tasks under resource constraints — it is now the default approach for most production fine-tuning workflows.