Abstract:Aligning generative models with human preference via RLHF typically suffers from overoptimization, where an imperfectly learned reward model can misguide the generative model to output undesired responses. We investigate this problem in a principled manner by identifying the source of the misalignment as a form of distributional shift and uncertainty in learning human preferences. To mitigate overoptimization, we first propose a theoretical algorithm that chooses the best policy for an adversarially chosen reward model; one that simultaneously minimizes the maximum likelihood estimation of the loss and a reward penalty term. Here, the reward penalty term is introduced to prevent the policy from choosing actions with spurious high proxy rewards, resulting in provable sample efficiency of the algorithm under a partial coverage style condition. Moving from theory to practice, the proposed algorithm further enjoys an equivalent but surprisingly easy-to-implement reformulation. Using the equivalence between reward models and the corresponding optimal policy, the algorithm features a simple objective that combines: (i) a preference optimization loss that directly aligns the policy with human preference, and (ii) a supervised learning loss that explicitly imitates the policy with a (suitable) baseline distribution. In the context of aligning large language models (LLM), this objective fuses the direct preference optimization (DPO) loss with the supervised fine-tuning (SFT) loss to help mitigate the overoptimization towards undesired responses, for which we name the algorithm Regularized Preference Optimization (RPO). Experiments of aligning LLMs demonstrate the improved performance of RPO compared with DPO baselines. Our work sheds light on the interplay between preference optimization and SFT in tuning LLMs with both theoretical guarantees and empirical evidence.

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Investigating on RLHF methodology

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Aligning Large Language Models with Human Preferences through Representation Engineering

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

Few-shot In-Context Preference Learning Using Large Language Models

More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness

Personalized Language Modeling from Personalized Human Feedback

Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective

Fine-Tuning Language Models with Advantage-Induced Policy Alignment

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

RRHF: Rank Responses to Align Language Models with Human Feedback

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

Active Preference Learning for Large Language Models

Stabilizing RLHF through Advantage Model and Selective Rehearsal