Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

Jian Li,Haojing Huang,Yujia Zhang,Pengfei Xu,Xi Chen,Rui Song,Lida Shi,Jingwen Wang,Hao Xu

2024-09-26

Abstract:Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at <a class="link-external link-https" href="https://github.com/lijian16/SPO" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

### The Problems Addressed by This Paper

This paper primarily addresses the following issues:

1. **Limitations of Binary Training Mechanisms**: Existing direct human preference alignment methods (such as Direct Preference Optimization, DPO) minimize or maximize loss on paired samples through a binary cross-entropy mechanism. However, this approach overlooks the variations in preference degrees among different responses. This results in large language models (LLMs) being unable to fully understand human preferences.

2. **Enhancing Preference Understanding**: To overcome the above limitations, the authors propose a new Self-supervised Preference Optimization (SPO) framework. This framework helps LLMs improve their ability to understand preference degrees by constructing a self-supervised preference degree loss that combines alignment loss.

3. **Simplifying the Data Annotation Process**: Traditional preference alignment methods require a large amount of manual annotation or ranking work. SPO, on the other hand, generates responses with different preference degrees by extracting key content from the outputs of LLMs and randomly removing this content, thus avoiding additional data collection and annotation work.

Through these improvements, SPO can significantly enhance the performance of existing preference alignment methods across different tasks and datasets, achieving state-of-the-art levels.