Abstract:Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quality rejected responses. The unawareness of the reward scores also drives the LLM to indiscriminately favor the low-quality chosen responses and fail to generalize to responses with the highest rewards, which are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset. This dataset is easily integrated with existing direct alignment algorithms and is applicable to any preference dataset. The experimental results across instruction-following benchmarks including AlpacaEval, MT-Bench, and Arena-Hard-Auto demonstrate that our approach consistently boosts the performance of DPO by a considerable margin across diverse models. Additionally, our method improves the average accuracy on various academic benchmarks. When applying our method to on-policy data, the resulting DPO model achieves SOTA results on AlpacaEval. Through ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere dataset expansion. Our code is available at <a class="link-external link-https" href="https://github.com/shenao-zhang/reward-augmented-preference" rel="external noopener nofollow">this https URL</a>.

Aligning LLMs with Domain Invariant Reward Models

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment

On Diversified Preferences of Large Language Model Alignment

Bayesian Reward Models for LLM Alignment

Transforming and Combining Rewards for Aligning Large Language Models

Prior Constraints-based Reward Model Training for Aligning Large Language Models

Aligning Crowd Feedback via Distributional Preference Reward Modeling

Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Confronting Reward Model Overoptimization with Constrained RLHF

Understanding the Learning Dynamics of Alignment with Human Feedback

HAF-RM: A Hybrid Alignment Framework for Reward Model Training

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation

Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning

Reward-Robust RLHF in LLMs