Abstract:Learning from preference-based feedback has recently gained traction as a promising approach to align language models with human interests. While these aligned generative models have demonstrated impressive capabilities across various tasks, their dependence on high-quality human preference data poses a bottleneck in practical applications. Specifically, noisy (incorrect and ambiguous) preference pairs in the dataset might restrict the language models from capturing human intent accurately. While practitioners have recently proposed heuristics to mitigate the effect of noisy preferences, a complete theoretical understanding of their workings remain elusive. In this work, we aim to bridge this gap by by introducing a general framework for policy optimization in the presence of random preference flips. We focus on the direct preference optimization (DPO) algorithm in particular since it assumes that preferences adhere to the Bradley-Terry-Luce (BTL) model, raising concerns about the impact of noisy data on the learned policy. We design a novel loss function, which de-bias the effect of noise on average, making a policy trained by minimizing that loss robust to the noise. Under log-linear parameterization of the policy class and assuming good feature coverage of the SFT policy, we prove that the sub-optimality gap of the proposed robust DPO (rDPO) policy compared to the optimal policy is of the order $O(\frac{1}{1-2\epsilon}\sqrt{\frac{d}{n}})$, where $\epsilon < 1/2$ is flip rate of labels, $d$ is policy parameter dimension and $n$ is size of dataset. Our experiments on IMDb sentiment generation and Anthropic's helpful-harmless dataset show that rDPO is robust to noise in preference labels compared to vanilla DPO and other heuristics proposed by practitioners.

Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

New Desiderata for Direct Preference Optimization

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Minor DPO reject penalty to increase training robustness

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

On the Generalization of Preference Learning with DPO

Direct Preference Optimization with an Offset

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Understanding Reference Policies in Direct Preference Optimization

Provably Robust DPO: Aligning Language Models with Noisy Feedback

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

$α$-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

$β$-DPO: Direct Preference Optimization with Dynamic $β$

sDPO: Don't Use Your Data All at Once

On Extending Direct Preference Optimization to Accommodate Ties

On Softmax Direct Preference Optimization for Recommendation

Direct Multi-Turn Preference Optimization for Language Agents

Uncertainty-Penalized Direct Preference Optimization