Prefix tuning is a parameter-efficient Fine-tuning method in which a small set of continuous, task-specific vectors - the "prefix" - is prepended to the key and value matrices inside each Transformer layer, while the underlying pre-trained weights are kept entirely frozen. Only the prefix parameters are updated during training, which can be as few as 0.1 % of the total model size, yet the prefix steers the model's Attention patterns toward the desired task behaviour. It was introduced by Li & Liang (2021) as an alternative to full fine-tuning that avoids catastrophic forgetting and allows one base model to serve many tasks by simply swapping prefix sets.

Think of it like adding a pair of prescription lenses in front of a camera lens without modifying the camera itself - the underlying optics stay identical, but what the sensor captures changes dramatically depending on which lens adapter you clip on.

To apply prefix tuning with Hugging Face PEFT:

from transformers import AutoModelForSeq2SeqLM
from peft import PrefixTuningConfig, get_peft_model, TaskType

config = PrefixTuningConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    num_virtual_tokens=20   # 20 learnable prefix tokens per layer
)
model = get_peft_model(
    AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large"),
    config
)
model.print_trainable_parameters()
# trainable params: 983,040 || all params: 783,933,440 (0.13 %)

Prefix tuning belongs to the same family as LoRA and QLoRA, all of which adapt large models without touching their core weights. It differs from prompt-based methods in Prompt Engineering because the prefix vectors live in embedding space and are not human-readable tokens. Understanding it also requires familiarity with Embedding representations and how Pretraining produces the frozen weights being steered.

You will encounter prefix tuning when reading research comparing parameter-efficient adaptation methods, or when a deployment team needs to serve one base LLM for dozens of tenants simultaneously - each tenant's lightweight prefix is loaded on demand without reloading the model.