Self-Improving Robust Preference Optimization

Eugene Choi,Arash Ahmadian,Matthieu Geist,Oilvier Pietquin,Mohammad Gheshlaghi Azar

2024-06-08

Abstract:Both online and offline RLHF methods such as PPO and DPO have been extremely successful in aligning AI with human preferences. Despite their success, the existing methods suffer from a fundamental problem that their optimal solution is highly task-dependent (i.e., not robust to out-of-distribution (OOD) tasks). Here we address this challenge by proposing Self-Improving Robust Preference Optimization SRPO, a practical and mathematically principled offline RLHF framework that is completely robust to the changes in the task. The key idea of SRPO is to cast the problem of learning from human preferences as a self-improvement process, which can be mathematically expressed in terms of a min-max objective that aims at joint optimization of self-improvement policy and the generative policy in an adversarial fashion. The solution for this optimization problem is independent of the training task and thus it is robust to its changes. We then show that this objective can be re-expressed in the form of a non-adversarial offline loss which can be optimized using standard supervised optimization techniques at scale without any need for reward model and online inference. We show the effectiveness of SRPO in terms of AI Win-Rate (WR) against human (GOLD) completions. In particular, when SRPO is evaluated on the OOD XSUM dataset, it outperforms the celebrated DPO by a clear margin of 15% after 5 self-revisions, achieving WR of 90%.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

This paper mainly discusses a problem in reinforcement learning from human feedback (RLHF) methods, which is that their optimal solutions depend heavily on the training task, leading to poor robustness to out-of-distribution tasks (OOD). To solve this problem, the paper proposes a "self-improving robust preference optimization" (SRPO) framework. SRPO transforms the problem learned from human preferences into a self-improvement process, which can be mathematically represented as a min-max optimization objective, aiming to jointly optimize self-improvement policy and generation policy. The solution to this optimization problem is independent of the training task, thus having robustness to task variations.

Specifically, SRPO consists of two steps: the first step is self-improvement preference optimization within the context, learning a model that can complete the task based on the input and recursively improve the output; the second step is to use the self-improvement policy learned in the first step to optimize the generation model and find the generation policy that requires the least improvement. Through these two steps, SRPO can perform large-scale optimization without reward models and online inference, using standard supervised optimization techniques.

Experimental results show that when evaluated on the OOD dataset XSum, SRPO outperforms DPO in terms of Win Rate, surpassing 90% after 5 self-revisions, with a 15% improvement.

In conclusion, this paper proposes a new method, SRPO, which addresses the instability issue in existing RLHF methods and improves the model's adaptability and generalization to task variations.