RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

Saeed Khaki,JinJin Li,Lan Ma,Liu Yang,Prathap Ramachandra

2024-03-31

Abstract:Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper addresses the efficiency and effectiveness issues when training large-scale language models to align with human intentions. Existing methods such as PPO, RS, and DPO have limitations in terms of instability, high resource consumption, or reliance on manual feedback. The paper proposes a new approach called RS-DPO, which combines rejection sampling (RS) and direct preference optimization (DPO). It first generates diverse responses through supervised fine-tuning, then selects contrasting sample pairs based on the reward distribution, and finally uses DPO for optimization. This method improves alignment performance, reduces resource requirements, and outperforms existing techniques in limited resource environments.

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

Statistical Rejection Sampling Improves Preference Optimization

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

Fine-Tuning Language Models with Advantage-Induced Policy Alignment

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

$α$-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs

DPO Meets PPO: Reinforced Token Optimization for RLHF

Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment

Accelerated Preference Optimization for Large Language Model Alignment

Towards Efficient Exact Optimization of Language Model Alignment

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

Aligning Large Language Models via Fine-grained Supervision

Direct Preference Optimization Using Sparse Feature-Level Constraints

Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts