TSO: Self-Training with Scaled Preference Optimization

Kaihui Chen,Hao Yi,Qingyang Li,Tianyu Qi,Yulan Hu,Fuzheng Zhang,Yong Liu
2024-08-31
Abstract:Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as attractive options due to offering effective improvement in simple, efficient, and stable without interactions with reward models. However, these offline preference optimization methods highly rely on the quality of pairwise preference samples. Meanwhile, numerous iterative methods require additional training of reward models to select positive and negative samples from the model's own generated responses for preference learning. Furthermore, as LLMs' capabilities advance, it is quite challenging to continuously construct high-quality positive and negative preference instances from the model's outputs due to the lack of diversity. To tackle these challenges, we propose TSO, or Self-Training with Scaled Preference Optimization, a framework for preference optimization that conducts self-training preference learning without training an additional reward model. TSO enhances the diversity of responses by constructing a model matrix and incorporating human preference responses. Furthermore, TSO introduces corrections for model preference errors through human and AI feedback. Finally, TSO adopts iterative and dual clip reward strategies to update the reference model and its responses, adaptively adjusting preference data and balancing the optimization process. Experimental results demonstrate that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing practical insight into preference data construction and model training strategies in the alignment domain.
Machine Learning,Artificial Intelligence,Computation and Language
What problem does this paper attempt to address?
The paper attempts to address the issue of how to improve the alignment of large language models (LLMs) with human preferences. Specifically: 1. **Quality Dependency in Preference Optimization**: Existing offline preference optimization methods (such as Direct Preference Optimization, DPO) heavily rely on the quality of paired preference samples. As the capabilities of LLMs increase, constructing high-quality positive and negative samples becomes increasingly difficult. 2. **Reward Model Requirement in Iterative Methods**: Many iterative methods require additional training of reward models to select positive and negative samples from model-generated responses for preference learning. 3. **Lack of Diversity and Adaptability**: Existing methods often focus only on the diversity and effectiveness of the data, neglecting adaptability. As the target model updates, its responses fail to update in a timely manner, leading to misleading signals. To address these challenges, the paper proposes TSO (Self-Training with Scaled Preference Optimization), a self-training preference learning framework that does not require additional training of reward models. TSO enhances response diversity by constructing a model matrix and incorporating human preference responses, and introduces methods to correct model preference errors through human and AI feedback. Additionally, TSO employs iterative and dual-clipping reward strategies to update the reference model and its responses, thereby dynamically adjusting preference data and balancing the optimization process. Experimental results show that TSO outperforms existing mainstream methods on various alignment evaluation benchmarks, providing important insights into preference data construction and model training strategies.