Abstract:We present a novel approach for enhancing diversity and control in data annotation tasks by personalizing large language models (LLMs). We investigate the impact of injecting diverse persona descriptions into LLM prompts across two studies, exploring whether personas increase annotation diversity and whether the impacts of individual personas on the resulting annotations are consistent and controllable. Our results show that persona-prompted LLMs produce more diverse annotations than LLMs prompted without personas and that these effects are both controllable and repeatable, making our approach a suitable tool for improving data annotation in subjective NLP tasks like toxicity detection.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to enhance diversity and control in data annotation tasks by personalizing large language models (LLMs). Specifically, the authors explore injecting different personas into LLM prompts to increase annotation diversity and verify whether these personas consistently and controllably affect annotation results. ### Main Research Questions 1. **RQ Study 1**: Does including persona descriptions in LLM prompts consistently increase the diversity of LLM annotation results? 2. **RQ Study 2**: Are there patterns in LLM annotations that align with the subjective observations of human annotators? ### Research Background Many natural language processing (NLP) tasks rely on data obtained through manual annotation, often collected via crowdsourcing studies. Röttger et al. (2022) introduced two paradigms for handling label differences caused by individual annotators' varying beliefs and backgrounds: - **Prescriptive Paradigm**: Suppresses annotators' subjectivity, aiming for a single label per data instance to make the training model more consistent. - **Descriptive Paradigm**: Encourages annotators' subjectivity, leveraging disagreements among annotators to improve the quality of model training and evaluation. ### Research Methods #### Data Sources - **Persona Descriptions**: From Chan et al. (2024)'s Persona Hub, containing 1 billion different persona descriptions. - **Toxicity Dataset**: Based on data prepared and shared by Sap et al. (2022), including 571 posts annotated by 173 annotators and 12 posts annotated by 641 annotators. #### Experimental Design - **Study 1**: Explores whether including persona descriptions in LLM prompts can increase the diversity of annotation results. The experiment uses two different open-source LLM models (Mistral-7B-Instruct-v0.1 and Qwen-7B-Instruct), conducting 1,000 annotation experiments with and without personas. - **Study 2**: Investigates whether annotation patterns among different personas align with the subjective patterns observed by human annotators. Through clustering analysis and correlation tests, it verifies whether the similarity of persona descriptions leads to similar annotation results. ### Main Findings - **Study 1**: LLM annotation results with persona descriptions show higher diversity, and this diversity is controllable. Different personas significantly influence LLM's annotation decisions. - **Study 2**: There is a positive correlation between the similarity of persona descriptions and the similarity of annotation results, indicating that controlling personas can guide LLM's annotation perspectives. ### Conclusion The proposed method successfully increases the diversity and control in data annotation by injecting personas into LLM prompts, making it suitable for NLP tasks that require consideration of subjectivity, such as toxicity detection.

Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

Editing Personality for LLMs

Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior

Quantifying the Persona Effect in LLM Simulations

On the steerability of large language models toward data-driven personas

ControlLM: Crafting Diverse Personalities for Language Models

Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization

Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

PERSONA: A Reproducible Testbed for Pluralistic Alignment

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

PersonalLLM: Tailoring LLMs to Individual Preferences

Which Demographics do LLMs Default to During Annotation?

Enhancing LLM-Based Human-Robot Interaction with Nuances for Diversity Awareness

PersLLM: A Personified Training Approach for Large Language Models

Unifying Data Perspectivism and Personalization: An Application to Social Norms

Extroversion or Introversion? Controlling The Personality of Your Large Language Models

Evaluating Large Language Model Biases in Persona-Steered Generation

BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data

Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agents

"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs