DPO (Direct Preference Optimization) is an alignment fine-tuning algorithm that adjusts a language model's behavior using human preference comparisons without requiring a separately trained reward model or a reinforcement learning optimizer. The method works by treating the language model itself as an implicit reward function: given paired responses to the same prompt — one preferred by annotators, one not — DPO updates the model to increase the relative likelihood of the preferred response while staying close to a frozen reference version of the model. The closeness constraint, controlled by a temperature-like beta parameter, prevents the model from over-optimizing toward surface patterns in the preference data.

Think of it like editing a draft by comparing it to a better version rather than against a rubric. Instead of scoring the draft on ten criteria and then rewriting based on scores, an editor simply marks which of two drafts is cleaner and the writer adjusts. No intermediate scoring step is needed.

In code, DPO training requires only a standard autoregressive model, a reference copy, and a preference dataset:

from trl import DPOTrainer, DPOConfig

config = DPOConfig(beta=0.1, learning_rate=5e-7, num_train_epochs=1)
trainer = DPOTrainer(
    model=policy_model,
    ref_model=reference_model,
    args=config,
    train_dataset=pref_dataset,
    tokenizer=tokenizer,
)
trainer.train()

The beta value near 0.1 is standard; values closer to 0 allow larger updates while values near 1 keep the model tightly anchored to its reference.

DPO is the primary alternative to RLHF for preference-based alignment and is often combined with LoRA for parameter-efficient Fine-tuning. The preference data format mirrors what RLHF reward models use, so organizations with existing RLHF pipelines can often reuse their datasets. Its simplicity has made it the default alignment step in most open-source instruction-tuning recipes.

You will encounter DPO in model training documentation, alignment research papers, and fine-tuning frameworks like TRL, LLaMA-Factory, and Axolotl.