Abstract:Reinforcement Learning from Human Feedback (RLHF) has proven highly effective in aligning Large Language Models (LLMs) with human preferences. However, the original RLHF typically optimizes under an overall reward, which can lead to a suboptimal learning process. This limitation stems from RLHF's lack of awareness regarding which specific tokens should be reinforced or suppressed. Moreover, conflicts in supervision can arise, for instance, when a chosen response includes erroneous tokens, while a rejected response contains accurate elements. To rectify these shortcomings, increasing dense reward methods, such as step-wise and token-wise RLHF, have been proposed. However, these existing methods are limited to specific tasks (like mathematics). In this paper, we propose the ``Adaptive Message-wise RLHF'' method, which robustly applies to various tasks. By defining pivot tokens as key indicators, our approach adaptively identifies essential information and converts sequence-level supervision into fine-grained, subsequence-level supervision. This aligns the density of rewards and action spaces more closely with the information density of the input. Experiments demonstrate that our method can be integrated into various training methods, significantly mitigating hallucinations and catastrophic forgetting problems, while outperforming other methods on multiple evaluation metrics. Our method improves the success rate on adversarial samples by 10\% compared to the sample-wise approach, and achieves a 1.3\% improvement on evaluation benchmarks such as MMLU, GSM8K, HumanEval, etc.

Automatic Successive Reinforcement Learning with Multiple Auxiliary Rewards.

Auxiliary Reward Generation with Transition Distance Representation Learning

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Intrinsically Motivated Self-supervised Learning in Reinforcement Learning

Reinforcement Learning with Automated Auxiliary Loss Search

Reinforcement Learning with Unsupervised Auxiliary Tasks

Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Deep Reinforcement Learning on Autonomous Driving Policy With Auxiliary Critic Network

Auxiliary task discovery through generate-and-test

Deep Reinforcement Learning for Autonomous Driving with an Auxiliary Actor Discriminator

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Hierarchical Multi-Agent Reinforcement Learning for Cooperative Tasks with Sparse Rewards in Continuous Domain

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment

Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays

Automated Reinforcement Learning: An Overview

Individual Reward Assisted Multi-Agent Reinforcement Learning.

Learning Dense Reward with Temporal Variant Self-Supervision

Multi-Objective Intrinsic Reward Learning for Conversational Recommender Systems

Decision-Focused Model-based Reinforcement Learning for Reward Transfer