Abstract:As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages real-time, in-situ user interactions to create preference datasets that more accurately reflect authentic human values. WildFeedback operates through a three-step process: feedback signal identification, preference data construction, and user-guided evaluation. We applied this framework to a large corpus of user-LLM conversations, resulting in a rich preference dataset that reflects genuine user preferences. This dataset captures the nuances of user preferences by identifying and classifying feedback signals within natural conversations, thereby enabling the construction of more representative and context-sensitive alignment data. Our extensive experiments demonstrate that LLMs fine-tuned on WildFeedback exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed user-guided evaluation. By incorporating real-time feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users. In summary, WildFeedback offers a robust, scalable solution for aligning LLMs with true human values, setting a new standard for the development and evaluation of user-centric language models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to better align large - language models (LLMs) with human preferences to ensure that their outputs are not only accurate but also in line with users' expectations and values. Specifically, existing alignment methods rely on manually - annotated datasets or synthetic datasets generated by LLMs, and these methods have problems such as being resource - intensive, highly subjective, and potentially amplifying model biases. To solve these problems, the author introduced a new framework - WILDFEEDBACK, which creates datasets that more truly reflect human preferences by leveraging real - time user - interaction feedback. ### Main Problems and Challenges 1. **Resource - Intensive**: Traditional methods rely on manually - annotated datasets, which require a great deal of manpower and time. 2. **Subjectivity**: Manually - annotated datasets may carry the personal preferences of the annotators and cannot fully represent the preferences of all users. 3. **Feedback Loop**: Using synthetic data generated by LLMs may cause the model's output to gradually be biased towards its own biases rather than diverse and real - user perspectives. 4. **Difficult to Scale**: Due to time and budget constraints, manually - annotated datasets are difficult to scale up on a large scale. ### WILDFEEDBACK's Solutions The WILDFEEDBACK framework solves the above problems through the following three steps: 1. **Feedback Signal Identification**: Automatically identify conversation segments containing feedback signals from natural conversations through user - satisfaction estimation techniques (such as SAT/DSAT scores). 2. **Preference Data Construction**: Based on the identified feedback signals, construct a preference dataset containing user preferences and corresponding responses. 3. **User - Guided Evaluation**: Propose a user - guided evaluation method to make model performance evaluation closer to the real - user preferences. ### Experimental Results Through extensive experiments, the author has proven that LLMs fine - tuned by WILDFEEDBACK show significantly better alignment effects in both traditional benchmark tests and the proposed user - guided evaluation framework. This marks a big step forward in developing more user - centered LLMs. ### Summary WILDFEEDBACK provides a powerful and scalable solution for aligning LLMs with real - human values, thereby increasing user satisfaction and setting new standards for the development and evaluation of user - centered language models.

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

Aligning LLMs with Individual Preferences via Interaction

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Constructive Large Language Models Alignment with Diverse Feedback

Your Weak LLM is Secretly a Strong Teacher for Alignment

Orchestrating LLMs with Different Personalizations

Understanding User Experience in Large Language Model Interactions

Aligning Large Language Models from Self-Reference AI Feedback with one General Principle

Understanding the Learning Dynamics of Alignment with Human Feedback

REAL: Response Embedding-based Alignment for LLMs

TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs

Reasons to Reject? Aligning Language Models with Judgments

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

Aligning Large Language Models with Human Preferences through Representation Engineering

Human-Instruction-Free LLM Self-Alignment with Limited Samples

Aligning Language Models with Offline Learning from Human Feedback

A Survey on Human Preference Learning for Large Language Models

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

UltraFeedback: Boosting Language Models with High-quality Feedback