Abstract:Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence. Its primary objective is to function as a human-centric (helpful, honest, and harmless) assistant. Alignment with humans assumes paramount significance, and reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit. Current technical routes usually include \textbf{reward models} to measure human preferences, \textbf{Proximal Policy Optimization} (PPO) to optimize policy model outputs, and \textbf{process supervision} to improve step-by-step reasoning capabilities. However, due to the challenges of reward design, environment interaction, and agent training, coupled with huge trial and error cost of large language models, there is a significant barrier for AI researchers to motivate the development of technical alignment and safe landing of LLMs. The stable training of RLHF has still been a puzzle. In the first report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training. We identify policy constraints being the key factor for the effective implementation of the PPO algorithm. Therefore, we explore the PPO-max, an advanced version of PPO algorithm, to efficiently improve the training stability of the policy model. Based on our main results, we perform a comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT. The absence of open-source implementations has posed significant challenges to the investigation of LLMs alignment. Therefore, we are eager to release technical reports, reward models and PPO codes, aiming to make modest contributions to the advancement of LLMs.

Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Learning to Synthesize Programs as Interpretable and Generalizable Policies

LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning

Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs

Secrets of RLHF in Large Language Models Part I: PPO

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

Large Language Models as Generalizable Policies for Embodied Tasks

Synthesizing Programmatic Policies with Actor-Critic Algorithms and ReLU Networks

Synthesizing Interpretable Control Policies through Large Language Model Guided Search

LGR2: Language Guided Reward Relabeling for Accelerating Hierarchical Reinforcement Learning

Code as Policies: Language Model Programs for Embodied Control

LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models

Synthesizing Programmatic Policy for Generalization Within Task Domain

RL-GPT: Integrating Reinforcement Learning and Code-as-policy

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

Reinforcement Learning Problem Solving with Large Language Models

Efficient Reinforcement Learning with Large Language Model Priors

Learning to Generate Better Than Your LLM