Direct Preference Optimization (DPO) is a fine-tuning algorithm that aligns a language model with human preferences by directly updating the model's parameters using paired comparison data, without training a separate reward model. Given a dataset of prompt-response pairs where each prompt has a preferred and a rejected completion, DPO derives a classification loss that increases the log-probability of preferred outputs relative to rejected ones, using the original reference model as an implicit reward baseline. This formulation collapses the three-stage RLHF pipeline — supervised fine-tuning, reward model training, and PPO optimization — into a single supervised pass.

Think of it like a teacher who, instead of scoring every essay and then coaching students based on the scores, simply shows each student two example essays and says "write more like this one, less like that one." The lesson is direct; no intermediate grading rubric is needed.

A DPO training loop with the TRL library looks like this:

from trl import DPOTrainer, DPOConfig

trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,
    args=DPOConfig(beta=0.1, max_length=512),
    train_dataset=preference_dataset,  # columns: prompt, chosen, rejected
    tokenizer=tokenizer,
)
trainer.train()

The beta hyperparameter controls how strongly the model is penalized for diverging from the reference policy; lower values allow more deviation.

DPO is an alternative to RLHF that trades flexibility for simplicity and stability — it avoids reward hacking and training instability associated with PPO. Fine-tuning on preference data is often combined with LoRA to reduce memory cost. The resulting model behavior is also shaped by the quality of the preference dataset, making data curation as important as the algorithm itself.

You will encounter DPO in alignment research and in open-source fine-tuning pipelines for instruction-following models, where it has largely replaced PPO-based RLHF as the default preference optimization method.