Abstract:Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice. In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77%. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models. Codes are available at <a class="link-external link-https" href="https://github.com/uclaml/SPPO" rel="external noopener nofollow">this https URL</a>.

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

Online Self-Preferring Language Models

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Self-Boosting Large Language Models with Synthetic Preference Data

Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization

SALMON: Self-Alignment with Instructable Reward Models

Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Aligning Large Language Models via Self-Steering Optimization

Self-Play Preference Optimization for Language Model Alignment

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment

Aligning Large Language Models with Self-generated Preference Data

Self-Evolved Reward Learning for LLMs

Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

Self-Rewarding Language Models

I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm

LIRE: listwise reward enhancement for preference alignment

SELU: Self-Learning Embodied MLLMs in Unknown Environments

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

Language Model Self-improvement by Reinforcement Learning Contemplation