Understanding the performance gap between online and offline alignment algorithms

Yunhao Tang,Daniel Zhaohan Guo,Zeyu Zheng,Daniele Calandriello,Yuan Cao,Eugene Tarassov,Rémi Munos,Bernardo Ávila Pires,Michal Valko,Yong Cheng,Will Dabney

2024-05-14

Abstract:Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper explores the performance gap between online and offline alignment algorithms in reinforcement learning. The research primarily focuses on the RLHF (Reinforcement Learning from Human Feedback) framework, which is used to improve large-scale language models' alignment. With the popularity of offline methods, researchers have started to question whether online interaction is necessary for AI alignment. The paper first demonstrates that, despite potential performance degradation in both methods due to over-optimization, online methods generally outperform offline methods. The authors then conducted a series of experiments to eliminate possible explanations, such as the coverage and quality of offline data, and found that while offline algorithms excel in classification tasks, they perform poorly in generation tasks, while online algorithms exhibit the opposite behavior. This indicates a unique interaction between discriminative and generative capabilities, influenced by the sampling process. The research also reveals that the performance gap exists in both comparative and non-comparative loss functions and cannot simply be resolved by scaling up the policy network. These findings highlight the crucial role of online sampling and imply the fundamental challenges of offline alignment algorithms. In summary, the paper attempts to address the question: why do online methods generally outperform offline methods in AI alignment? Through experiments, the research reveals that the performance differences between the two may be attributed to the combined effects of data diversity, algorithmic optimization characteristics, loss functions, and model scale.

Understanding the performance gap between online and offline alignment algorithms

Beyond Reward: Offline Preference-guided Policy Optimization

Behavior Proximal Policy Optimization

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Aligning Language Models with Offline Learning from Human Feedback

Learn Your Reference Model for Real Good Alignment

Actor-Critic Alignment for Offline-to-Online Reinforcement Learning

Offline Regularised Reinforcement Learning for Large Language Models Alignment

The Real, the Better: Aligning Large Language Models with Online Human Behaviors

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Is Value Learning Really the Main Bottleneck in Offline RL?

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

Human Alignment of Large Language Models through Online Preference Optimisation

Fine-Tuning Language Models with Advantage-Induced Policy Alignment

Interpretable performance analysis towards offline reinforcement learning: A dataset perspective

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning