Training
DPO
Direct Preference Optimization — a simpler alternative to RLHF that fine-tunes a model directly on preference pairs, no separate reward model required.
01 ——
In plain English
DPO (Direct Preference Optimization) is a training method for aligning a language model with human preferences. Like RLHF, it learns from comparisons ("response A is better than response B"). Unlike RLHF, it skips the intermediate step of training a reward model and instead optimises the language model directly.
Why teams use DPO:
- Simpler — fewer moving parts than full RLHF
- Cheaper — no separate reward-model training pass
- More stable — avoids the reward-hacking and instability common in RL
- Reproducible — easier to debug and re-run
Limits:
- Less expressive than RLHF for complex multi-step preferences
- Needs clean preference data (noisy pairs hurt it more than RLHF)
- Often combined with SFT (supervised fine-tuning) for best results
Variants: KTO (Kahneman-Tversky Optimization), IPO, ORPO, and SimPO are all DPO descendants. Many open-source instruction-tuned models (Llama-Instruct, Qwen-Instruct) use DPO or a variant for the final alignment pass.
02 ——
Related terms
RLHF
Reinforcement Learning from Human Feedback — the training technique that teaches AI models to be helpful, harmless, and honest.
RLAIF
Reinforcement Learning from AI Feedback — alignment training where another AI model, not a human, provides the preference signal used to fine-tune the target model.
Fine-tuning
Further training a pre-trained AI model on your own data to specialise it for a specific task or style.
Post-training
Everything done to a model after pretraining — fine-tuning, RLHF, DPO, safety training — to turn a raw base model into a usable product.
Alignment
The challenge of making AI systems behave in ways that match human values and intentions — not just their literal instructions.
Training Data
The dataset an AI model learns from — its quality, diversity, and biases directly shape what the model can do and how well it does it.
Back to glossaryLast reviewed May 2026