Abstract:The importance of Reinforcement Learning from Human Feedback (RLHF) in aligning large language models (LLMs) with human values cannot be overstated. RLHF is a three-stage process that includes supervised fine-tuning (SFT), reward learning, and policy learning. Although there are several offline and online approaches to aligning LLMs, they often suffer from distribution shift issues. These issues arise from the inability to accurately capture the distributional interdependence between the reward learning and policy learning stages. Consequently, this has led to various approximated approaches, but the theoretical insights and motivations remain largely limited to tabular settings, which do not hold in practice. This gap between theoretical insights and practical implementations is critical. It is challenging to address this gap as it requires analyzing the performance of AI alignment algorithms in neural network-parameterized settings. Although bi-level formulations have shown promise in addressing distribution shift issues, they suffer from the hyper-gradient problem, and current approaches lack efficient algorithms to solve this. In this work, we tackle these challenges employing the bi-level formulation laid out in Kwon et al. (2024) along with the assumption \emph{Weak Gradient Domination} to demonstrate convergence in an RLHF setup, obtaining a sample complexity of $\epsilon^{-\frac{7}{2}}$ . Our key contributions are twofold: (i) We propose a bi-level formulation for AI alignment in parameterized settings and introduce a first-order approach to solve this problem. (ii) We analyze the theoretical convergence rates of the proposed algorithm and derive state-of-the-art bounds. To the best of our knowledge, this is the first work to establish convergence rate bounds and global optimality for the RLHF framework in neural network-parameterized settings.

The Correct Convergence of the Rival Penalized Competitive Learning (rpcl) Algorithm

Convergence Analysis of Rival Penalized Competitive Learning (RPCL) Algorithm

A Cost-Function Approach to Rival Penalized Competitive Learning (RPCL)

The mahalanobis distance based rival penalized competitive learning algorithm

Convergence Analysis of an Incremental Approach to Online Inverse Reinforcement Learning

Convergence Rate of Primal-Dual Approach to Constrained Reinforcement Learning with Softmax Policy

Convergence Theorems for a Class of Learning Algorithms with VLRPs

Deep Rival Penalized Competitive Learning for Low-resolution Face Recognition

Application of Dynamic Rival Penalized Competitive Learning on the Clustering Analysis of Seismic Data.

BYY Harmony Learning on Finite Mixture: Adaptive Gradient Implementation and A Floating RPCL Mechanism

Convergence Theorems for the Kohonen Feature Mapping Algorithms with VLRPs

On The Global Convergence Of Online RLHF With Neural Parametrization

Convergence of Unregularized Online Learning Algorithms

Fast and Strong Convergence of Online Learning Algorithms.

Global Convergence of Receding-Horizon Policy Search in Learning Estimator Designs

A Tighter Convergence Proof of Reverse Experience Replay

A Fast Accurate Two-stage Training Algorithm for L1-regularized CRFs with Heuristic Line Search Strategy.

A Comparative Study of RPCL and MCE Based Discriminative Training Methods for LVCSR.

On Convergence Rates of Robust Adaptive Game Theoretic Learning Algorithms

High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noise

Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness