Abstract:Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

Reward Generalization in RLHF: A Topological Perspective

Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective

Prototypical Reward Network for Data-Efficient RLHF

Secrets of RLHF in Large Language Models Part II: Reward Modeling

Prototypical Reward Network for Data-Efficient Model Alignment

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

How to Evaluate Reward Models for RLHF

R3HF: Reward Redistribution for Enhancing Reinforcement Learning from Human Feedback

RRHF: Rank Responses to Align Language Models with Human Feedback

Reward-Robust RLHF in LLMs

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment

Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

Confronting Reward Model Overoptimization with Constrained RLHF

MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences

Personalized Language Modeling from Personalized Human Feedback

ALaRM: Align Language Models via Hierarchical Rewards Modeling

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation