Abstract:Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these differences arise, traditional RLHF frameworks simply average over them, leading to inaccurate rewards and poor performance for individual subgroups. To address the need for pluralistic alignment, we develop a class of multimodal RLHF methods. Our proposed techniques are based on a latent variable formulation - inferring a novel user-specific latent and learning reward models and policies conditioned on this latent without additional user-specific data. While conceptually simple, we show that in practice, this reward modeling requires careful algorithmic considerations around model architecture and reward scaling. To empirically validate our proposed technique, we first show that it can provide a way to combat underspecification in simulated control problems, inferring and optimizing user-specific reward functions. Next, we conduct experiments on pluralistic language datasets representing diverse user preferences and demonstrate improved reward function accuracy. We additionally show the benefits of this probabilistic framework in terms of measuring uncertainty, and actively learning user preferences. This work enables learning from diverse populations of users with divergent preferences, an important challenge that naturally occurs in problems from robot learning to foundation model alignment.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to better adapt to the diverse preferences of different users in Reinforcement Learning from Human Feedback (RLHF). Traditional RLHF methods assume that all users' preferences come from the same utility function, which leads to the model being unable to accurately capture the unique needs of individual users when dealing with user groups with different preferences, thus affecting performance and accuracy. Specifically: 1. **Problem of unimodal preference assumption**: Current RLHF techniques usually adopt the unimodal assumption, that is, all people's preferences can be explained by a single utility function. However, in reality, there are significant preference differences among different users, and these differences will cause the model to average these preferences during the optimization process, so it cannot accurately reflect the preferences of specific users. 2. **Preferences of minority groups are ignored**: When most users have a weak preference for a certain result, while a small number of users strongly oppose this result, the existing RLHF methods tend to ignore the preferences of minority groups, resulting in a decline in model performance. 3. **Multimodal preference modeling**: To address the above problems, the paper proposes a multimodal preference learning method based on variational inference (Variational Preference Learning, VPL). This method introduces a latent variable \( z \) to represent the hidden context of users, and on this basis, learns the conditional reward model and policy, so that it can more accurately capture and adapt to the diverse preferences of different users. ### Main contributions of the paper 1. **Multimodal preference modeling**: Proposed a multimodal preference learning framework based on latent variables, which can effectively handle the diverse preferences of different users. 2. **Latent variable inference**: Through the variational inference method, infer the latent variable distribution of users from a small number of user preference labels, thereby achieving personalized reward modeling. 3. **Reward function scaling**: Proposed a reward function scaling method based on preference likelihood, which solves the problem of inconsistent reward scales in multi - task learning. 4. **Active learning**: Utilize the probabilistic characteristics of the variational encoder to design an active learning strategy, and minimize the uncertainty of latent variables by selecting the most informative query set. ### Experimental verification The paper verifies the effectiveness of the VPL method through experiments on multiple simulated control tasks and language tasks: 1. **Simulated control tasks**: In multiple simulated environments, VPL can accurately model multimodal reward functions and improve task performance and personalization ability. 2. **Language tasks**: In handling language tasks with conflicting user preferences, the multimodal reward model trained by VPL can predict rewards more accurately and improve model performance. In general, this paper aims to solve the deficiencies of traditional RLHF methods in dealing with diverse user preferences. By introducing latent variables and variational inference methods, more accurate and personalized reward modeling is achieved.

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Personalized Language Modeling from Personalized Human Feedback

Reinforcement Learning from Diverse Human Preferences

Multi-turn Reinforcement Learning from Preference Human Feedback

The History and Risks of Reinforcement Learning and Human Feedback

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Provable Multi-Party Reinforcement Learning with Diverse Human Feedback

REvolve: Reward Evolution with Large Language Models using Human Feedback

On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Optimal Design for Reward Modeling in RLHF

Models of human preference for learning reward functions

Few-shot In-Context Preference Learning Using Large Language Models

Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization

Fine-Tuning Language Models with Reward Learning on Policy

Sample Efficient Reinforcement Learning from Human Feedback via Active Exploration

Dual Active Learning for Reinforcement Learning from Human Feedback