中文版

DPO paper: https://arxiv.org/pdf/2305.18290

DPO 算法详解：从理论到实现

1. 什么是 DPO？

DPO（Direct Preference Optimization）是一种直接基于人类偏好进行优化的算法，旨在解决从人类偏好数据中训练出表现更优的语言模型的问题。它与传统的基于奖励建模的强化学习方法（如 PPO）不同，通过引入一种基于 Bradley-Terry 模型的参数化方法，将人类偏好概率直接与语言模型的输出概率相关联，从而避免了明确训练奖励模型的过程。

---

2. DPO 解决什么问题？

在 RLHF（Reinforcement Learning with Human Feedback）框架中，通常需要训练一个奖励模型来对语言模型的生成进行打分。然而，训练奖励模型和使用强化学习优化策略模型（如 PPO）通常会引入一些复杂性和不稳定性：

• 奖励模型可能过拟合或偏离人类真实偏好。
• 使用强化学习优化策略模型需要平衡探索和收敛，容易引发 KL 散度爆炸等问题。

DPO 提供了一种更直接的优化方式，通过重新参数化，将偏好建模直接嵌入语言模型优化中，从而绕过奖励建模，简化了训练流程。

---

3. DPO 的核心公式

DPO 的核心思想是通过 Bradley-Terry 偏好模型，将偏好概率建模为语言模型输出概率的对数比值，并引入温度参数 ( $\beta$ ) 来控制 KL 惩罚强度。

核心公式

人类偏好概率建模公式如下：

$$p^{\ast }(y_1\succ y_2|x)=\frac{1}{1+\exp \left(\beta \log \frac{{\pi }^{\ast }(y_2|x)}{{\pi }_{\text{ref}}(y_2|x)}-\beta \log \frac{{\pi }^{\ast }(y_1|x)}{{\pi }_{\text{ref}}(y_1|x)}\right)}$$

在实际中，我们通过最大化以下目标函数来优化参数化的策略模型 ( ${\pi }_{\theta }$ )：

$$L_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\log \sigma \left(\beta \log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{\text{ref}}(y_w|x)}-\beta \log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{\text{ref}}(y_l|x)}\right)\right]$$

其中：

• ( $\sigma$ ) 是 Sigmoid 函数。
• ( $y_w$ ) 和 ( $y_l$ ) 分别是人类标注的偏好和非偏好样本。

通过最大化该目标函数，策略模型会更倾向于生成被人类偏好的输出，同时抑制被人类不喜欢的输出。

---

4. 如何理解 DPO？

DPO 的优化过程可以从以下几个方面理解：

1. 奖励重新参数化
   通过将奖励模型嵌入策略模型输出的对数比值中，避免了显式训练奖励模型的过程。
   隐式奖励定义为：
   $${\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{\text{ref}}(y|x)}$$

2. 梯度优化
   DPO 的梯度公式为：
   $${\nabla }_{\theta }{L}_{\text{DPO}}=-\beta {\mathbb{E}}_{(x,y_w,y_l)\sim D}\left[\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))\cdot ({\nabla }_{\theta }\log {\pi }_{\theta }(y_w|x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l|x))\right]$$

   直观上，这意味着模型会：

   • 提高 ( $y_w$ ) 的生成概率。
   • 降低 ( $y_l$ ) 的生成概率。
   • 偏差较大的样本（即 ( ${\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w)$ ) 较大时）权重更高。

3. 温度参数 ( $\beta$ )
   ( $\beta$ ) 控制 KL 惩罚的强度，平衡策略模型与参考模型之间的分布差异。

---

5. 示例解析

假设我们有一个 Prompt，生成了两个候选回复 ( $y_1$ ) 和 ( $y_2$ )，并根据人类偏好得到以下信息：

• ( $y_1$ ) 被偏好 (( $y_w=y_1$ ))，( $y_2$ ) 被不偏好 (( $y_l=y_2$ ))。
• 模型的输出概率为：
  $${\pi }_{\theta }(y_1|x)=0.6,{\pi }_{\theta }(y_2|x)=0.4,{\pi }_{\text{ref}}(y_1|x)=0.5,{\pi }_{\text{ref}}(y_2|x)=0.5$$

计算隐式奖励：
$${\hat{r}}_{\theta }(x,y_1)=\beta \log \frac{{\pi }_{\theta }(y_1|x)}{{\pi }_{\text{ref}}(y_1|x)}=\beta \log \frac{0.6}{0.5}$$
$${\hat{r}}_{\theta }(x,y_2)=\beta \log \frac{{\pi }_{\theta }(y_2|x)}{{\pi }_{\text{ref}}(y_2|x)}=\beta \log \frac{0.4}{0.5}$$

偏好模型的概率：
$$p^{\ast }(y_1\succ y_2|x)=\frac{1}{1+\exp \left({\hat{r}}_{\theta }(x,y_2)-{\hat{r}}_{\theta }(x,y_1)\right)}$$

优化目标是让模型进一步增加 ( $y_1$ ) 的概率，同时减少 ( $y_2$ ) 的概率。

---

6. DPO 和 PPO 的区别

特性	DPO	PPO
核心思想	直接基于人类偏好优化语言模型	基于奖励信号，通过强化学习优化策略
是否需要奖励模型	不需要	需要
优化目标	最大化偏好概率	最大化累计奖励
实现复杂度	较低	较高
稳定性	较高	可能出现 KL 爆炸等问题

关于KL爆炸问题，可以参考笔者的另一篇博客：PPO 可能出现 KL 爆炸等问题的详细分析(KL Explosions in PPO)： 中英双语

---

7. 总结

DPO 提供了一种高效、稳定的语言模型优化方法，适合在大规模人类偏好数据上训练更优的模型。相比于传统的 RLHF 方法，DPO 不仅简化了实现过程，还具备更强的理论一致性和实践可靠性。

Direct Preference Optimization (DPO): A Comprehensive Overview

What Problem Does DPO Solve?

Direct Preference Optimization (DPO) addresses the limitations of Reinforcement Learning with Human Feedback (RLHF) by offering a simpler and more direct optimization method. RLHF traditionally uses reward models and Proximal Policy Optimization (PPO) to align language models with human preferences. However, PPO introduces complexity due to the need for dynamic reward modeling and reinforcement learning updates, which involve policy rollouts and value function estimation.

DPO simplifies this process by directly optimizing the likelihood of human-preferred responses relative to dispreferred ones without requiring an explicit reward model or reinforcement learning steps. Instead, it reformulates the optimization as a maximum likelihood estimation (MLE) problem.

Core Formula of DPO

The central idea of DPO is to use a Bradley-Terry preference model to define probabilities for human preferences based on the log-probabilities output by the model.

Given:

• ( ${\pi }_{\theta }$ ): The policy (current model being optimized)
• ( ${\pi }_{ref}$ ): The reference policy (pre-trained model used as a baseline)
• ( $y_w$ ): Preferred response
• ( $y_l$ ): Dispreferred response
• ( $\beta$ ): Temperature hyperparameter controlling regularization strength

DPO models human preferences using the log-ratio of probabilities between the preferred and dispreferred outputs.

The loss function is:
$$L_{DPO}({\pi }_{\theta };{\pi }_{ref})=-E_{(x,y_w,y_l)\sim D}\left[\log \sigma \left(\beta \left(\log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}-\log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)}\right)\right)\right]$$

Key Points in the Formula:

1. The loss directly optimizes the relative log-probabilities of preferred (($y_w$)) versus dispreferred (($y_l$)) responses.
2. ( $\beta$ ) controls the strength of KL-regularization between the policy and the reference model.
3. ( $\sigma (\cdot )$ ) represents the sigmoid function, ensuring the preference probabilities are modeled effectively.
4. It eliminates the need for explicit reward modeling, treating model preferences as implicit rewards.

Understanding the Formula

1. Implicit Reward Calculation

DPO implicitly defines a reward function based on the policy and reference model:

$${\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{ref}(y|x)}$$

This means the reward is proportional to the log-likelihood ratio between the current and reference models.

2. Optimization Objective

DPO optimizes the probability of preferred completions being ranked higher than dispreferred completions.

Specifically, it increases the likelihood of preferred completions (($y_w$)) while decreasing the likelihood of dispreferred ones (($y_l$)).

The gradient of the loss is:
$${\nabla }_{\theta }{L}_{DPO}=-\beta E_{(x,y_w,y_l)\sim D}\left[\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))\left({\nabla }_{\theta }\log {\pi }_{\theta }(y_w|x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l|x)\right)\right]$$

3. Weighting by Confidence

The weighting term ( $\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))$ ) penalizes errors when the model incorrectly assigns higher rewards to dispreferred completions. This ensures that updates focus on examples where the model is most uncertain or wrong, leading to more effective training.

---

Example Analysis

Suppose we have the following preferences for prompts:

Input Prompt:
“What is the capital of France?”

Completions:

• ( $y_w$ ): “The capital of France is Paris.” (Preferred)
• ( $y_l$ ): “The capital of France is London.” (Dispreferred)

The log-probabilities from the current model (( ${\pi }_{\theta }$ )) and reference model (( ${\pi }_{ref}$ )) are:

• ( ${\pi }_{\theta }(y_w|x)=-0.2$ ), ( ${\pi }_{\theta }(y_l|x)=-0.8$ )
• ( ${\pi }_{ref}(y_w|x)=-0.3$ ), ( ${\pi }_{ref}(y_l|x)=-0.7$ )

Using the DPO loss formula:

1. Calculate the log-probability ratios:
   $$r_w=\log \frac{{\pi }_{\theta }(y_w|x)}{{\pi }_{ref}(y_w|x)}=\log (-0.2)-\log (-0.3)=-0.17$$
   $$r_l=\log \frac{{\pi }_{\theta }(y_l|x)}{{\pi }_{ref}(y_l|x)}=\log (-0.8)-\log (-0.7)=0.06$$

2. Compute the preference difference:
   $$\Delta r=\beta (r_w-r_l)=\beta (-0.17-0.06)=\beta (-0.23)$$

3. Final loss:
   $$L=-\log \sigma (\Delta r)=-\log \sigma (-0.23\beta )$$

The optimization encourages increasing the likelihood of ( $y_w$ ) while reducing ( $y_l$ ).

---

DPO vs PPO: Key Differences

Aspect	DPO	PPO
Reward Model	Implicitly modeled via log-probabilities.	Requires an explicit, learned reward model.
Algorithm Type	Maximum Likelihood Estimation (MLE).	Reinforcement Learning with Policy Gradients.
Training Complexity	Simpler and requires fewer hyperparameters.	More complex with value function updates and clipping mechanisms.
Stability	More stable due to direct optimization.	Requires careful tuning to avoid divergence.
Data Requirement	Relies on preference data directly.	Requires preference data and rollout data for updates.
KL Regularization	Controlled by parameter ( $\beta$ ).	Controlled by PPO clipping.

---

Why is DPO Effective?

1. Simplified Training Process: No need for reward model training or complex PPO pipelines.
2. Implicit Reward Modeling: Avoids separate reward models and leverages pre-trained probabilities.
3. Theoretical Guarantees: Based on Bradley-Terry models, ensuring consistency under reasonable assumptions.
4. Practical Applicability: Compatible with public preference datasets without requiring new data collection.

---

Implementation Example

import torch
import torch.nn.functional as F

def dpo_loss(pi_logps, ref_logps, yw_idxs, yl_idxs, beta):
    pi_yw_logps, pi_yl_logps = pi_logps[yw_idxs], pi_logps[yl_idxs]
    ref_yw_logps, ref_yl_logps = ref_logps[yw_idxs], ref_logps[yl_idxs]
    pi_logratios = pi_yw_logps - pi_yl_logps
    ref_logratios = ref_yw_logps - ref_yl_logps
    losses = -F.logsigmoid(beta * (pi_logratios - ref_logratios))
    rewards = beta * (pi_logps - ref_logps).detach()
    return losses, rewards

---

Conclusion

DPO offers a lightweight alternative to PPO for preference optimization by directly leveraging preference data without relying on complex reinforcement learning frameworks. It is particularly effective for aligning language models with human preferences and offers theoretical guarantees grounded in Bradley-Terry models. Given its simplicity and effectiveness, DPO is increasingly used for tasks requiring preference-based fine-tuning of large language models.

后记

2024年12月26日20点52分于上海，在GPT4o大模型辅助下完成。