Abstract:Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

Human irrationality: both bad and good for reward inference

On the Sensitivity of Reward Inference to Misspecified Human Models

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Reward Modeling Requires Automatic Adjustment Based on Data Quality

Ongoing, rational calibration of reward-driven perceptual biases

Models of human preference for learning reward functions

When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback

Active Reward Learning from Multiple Teachers

Limited or Biased: Modeling Sub-Rational Human Investors in Financial Markets

Scaling Laws for Reward Model Overoptimization

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models

Mitigating Reward Hacking Via Information-Theoretic Reward Modeling.

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?

The History and Risks of Reinforcement Learning and Human Feedback

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Confronting Reward Model Overoptimization with Constrained RLHF

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling