DPO (Direct Preference Optimization) is a supervised fine-tuning algorithm that teaches a language model human preferences by training directly on pairs of outputs — one preferred, one rejected — for the same prompt. Unlike earlier reinforcement learning approaches, DPO does not require training a separate reward model or running a reinforcement learning loop; it reparameterizes the reward function in terms of the model's own log-probabilities relative to a frozen reference copy, yielding a standard binary cross-entropy objective.

Think of it like learning table manners by being shown two meals side by side — "eat like this, not like that" — rather than having an etiquette judge score every bite and report back before you adjust. The feedback is direct and the correction is immediate.

The preference dataset has a simple structure:

// Each row contains a prompt + two completions
{
  "prompt": "Explain quantum entanglement.",
  "chosen": "Quantum entanglement is a phenomenon where...",
  "rejected": "Its like when two particles become friends..."
}

Training maximizes the margin between the log-probabilities of chosen and rejected completions, weighted by how confident the reference model is in each.

DPO is typically applied after an initial supervised Fine-tuning phase and is positioned as a simpler alternative to RLHF. It shares the goal of making LLMs more helpful and less harmful, and the preference data used to train it is similar to the data that powers RLHF reward models. Variants like IPO and KTO extend the idea to different loss formulations.

You will encounter DPO in model cards and alignment papers for most instruction-tuned open-source models released since 2023, and in fine-tuning libraries like TRL and Axolotl.