Abstract:Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice. In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77%. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models. Codes are available at <a class="link-external link-https" href="https://github.com/uclaml/SPPO" rel="external noopener nofollow">this https URL</a>.

Multi-objective Reinforcement learning from AI Feedback

Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignment

MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

A Critical Evaluation of AI Feedback for Aligning Large Language Models

Fine-Tuning Language Models with Advantage-Induced Policy Alignment

Reinforcement Learning from AI Feedback A Review

Multi-turn Reinforcement Learning from Preference Human Feedback

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning

Self-Play Preference Optimization for Language Model Alignment

LIRE: listwise reward enhancement for preference alignment

Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences

Demonstration Guided Multi-Objective Reinforcement Learning

Continual Multi-Objective Reinforcement Learning Via Reward Model Rehearsal

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

Offline Regularised Reinforcement Learning for Large Language Models Alignment

C-MORL: Multi-Objective Reinforcement Learning through Efficient Discovery of Pareto Front

Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AI

Aligning Large Language Models via Fine-grained Supervision