DPO
Direct Preference Optimization — an RLHF-like fine-tuning approach that skips training an explicit reward model, instead directly optimizing the policy against pairwise preferences. Simpler and often more stable than PPO-based RLHF; widely adopted in open-weights fine-tuning.
Related terms
- RLHF
Reinforcement Learning from Human Feedback — a training method where human preferences over pairs of outputs are used to train a reward model, which in turn shapes the base LLM via RL (typically PPO or DPO). Central to aligning modern chat assistants.
- Fine-tuning
Continuing to train a pretrained model on a task-specific dataset to change its behavior, style, or domain knowledge. Techniques range from full fine-tuning (all weights) to parameter-efficient methods like LoRA and QLoRA that update only small adapter layers.
- Alignment
The general problem of making an AI system behave according to its designers' or operators' intent, especially as capability grows. Practical alignment work spans RLHF, Constitutional AI, red-teaming, evaluation, and interpretability research.
Related on this site
Where this fits
DPO is part of the AI Fundamentals vocabulary used in the Generative AI Maturity Framework. See the full glossary for the complete set of 149 defined terms, or take the free maturity assessment to see where your organisation stands.