Abstract:Persona assignment has become a common strategy for customizing LLM use to particular tasks and contexts. In this study, we explore how evaluation of different nations change when LLMs are assigned specific nationality personas. We assign 193 different nationality personas (e.g., an American person) to four LLMs and examine how the LLM evaluations (or ''perceptions'')of countries change. We find that all LLM-persona combinations tend to favor Western European nations, though nation-personas push LLM behaviors to focus more on and treat the nation-persona's own region more favorably. Eastern European, Latin American, and African nations are treated more negatively by different nationality personas. We additionally find that evaluations by nation-persona LLMs of other nations correlate with human survey responses but fail to match the values closely. Our study provides insight into how biases and stereotypes are realized within LLMs when adopting different national personas. In line with the ''Blueprint for an AI Bill of Rights'', our findings underscore the critical need for developing mechanisms to ensure that LLM outputs promote fairness and avoid over-generalization.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper explores how the evaluation of different countries changes when large language models (LLMs) are assigned specific national identities. Specifically, the researchers assigned 193 different national identities to four LLMs and analyzed the behavior and evaluation of these models under different national identities. The main research questions include: 1. **How does national identity affect the "perception" or evaluation of different countries by large language models?** 2. **What bias patterns are exhibited at the regional level in the content generated by LLMs after applying national identities?** 3. **To what extent does the content generated by LLMs with national identities match human survey data on national perceptions?** ### Main Findings - **Pervasive Western Bias**: Regardless of the assigned national identity, all LLMs showed a preference for Western countries, especially Western European countries. - **Significant Impact of National Identity on Response Frequency**: After assigning national identities, the models were more inclined to focus on other countries in the same region, but the impact on which countries were evaluated positively or negatively was minimal. - **Correlation with Human Survey Data**: The content generated by LLMs with national identities correlated with human survey data from the corresponding countries, but the absolute values did not completely match. Notably, the LLMs most closely matched the survey data patterns of the United States. ### Research Significance This study reveals how large language models exhibit biases and stereotypes when adopting different national identities, highlighting the need to develop mechanisms to ensure that LLM outputs promote fairness and avoid overgeneralization. This aligns with the principles outlined in the "Blueprint for an AI Bill of Rights," which states that automated systems should uphold principles of fairness, transparency, and accountability.

Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

"A Woman is More Culturally Knowledgeable than A Man?": The Effect of Personas on Cultural Norm Interpretation in LLMs

Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

LLMs' ways of seeing User Personas

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

Evaluation of LLMs Biases Towards Elite Universities: A Persona-Based Exploration

Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas

Personas with Attitudes: Controlling LLMs for Diverse Data Annotation

Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization

PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits

"Global is Good, Local is Bad?": Understanding Brand Bias in LLMs

Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates

Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models

Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

Persona Setting Pitfall: Persistent Outgroup Biases in Large Language Models Arising from Social Identity Adoption

Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agents

Hate Personified: Investigating the role of LLMs in content moderation

Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles

Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

How Well Do LLMs Represent Values Across Cultures? Empirical Analysis of LLM Responses Based on Hofstede Cultural Dimensions

PersonalLLM: Tailoring LLMs to Individual Preferences