Abstract:Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homogeneous, and can be encoded by a single reward model. In this paper, we focus on addressing the issues due to the inherent heterogeneity in human preferences, as well as their potential strategic behavior in providing feedback. Specifically, we propose two frameworks to address heterogeneous human feedback in principled ways: personalization-based one and aggregation-based one. For the former, we propose two approaches based on representation learning and clustering, respectively, for learning multiple reward models that trades off the bias (due to preference heterogeneity) and variance (due to the use of fewer data for learning each model by personalization). We then establish sample complexity guarantees for both approaches. For the latter, we aim to adhere to the single-model framework, as already deployed in the current RLHF paradigm, by carefully aggregating diverse and truthful preferences from humans. We propose two approaches based on reward and preference aggregation, respectively: the former utilizes both utilitarianism and Leximin approaches to aggregate individual reward models, with sample complexity guarantees; the latter directly aggregates the human feedback in the form of probabilistic opinions. Under the probabilistic-opinion-feedback model, we also develop an approach to handle strategic human labelers who may bias and manipulate the aggregated preferences with untruthful feedback. Based on the ideas in mechanism design, our approach ensures truthful preference reporting, with the induced aggregation rule maximizing social welfare functions.

AGR: Age Group fairness Reward for Bias Mitigation in LLMs

Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems

Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs

What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning

Balancing the Scales: Reinforcement Learning for Fair Classification

UP5: Unbiased Foundation Model for Fairness-aware Recommendation

Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning

Adaptive Priority Reweighing for Generalizing Fairness Improvement.

Retiring Adult: New Datasets for Fair Machine Learning

A Study of Implicit Ranking Unfairness in Large Language Models

Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era

On Fairness of Low-Rank Adaptation of Large Models

Low-rank finetuning for LLMs: A fairness perspective

Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographics

Towards Harmless Rawlsian Fairness Regardless of Demographic Prior

A Normative Framework for Benchmarking Consumer Fairness in Large Language Model Recommender System

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

A Survey on Fairness in Large Language Models

Alpha and Prejudice: Improving $α$-sized Worst-case Fairness via Intrinsic Reweighting

On the Fairness of Randomized Trials for Recommendation with Heterogeneous Demographics and Beyond

No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users