What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages

Viktor Mihaylov,Aleksandar Shtedritski

2024-07-13

Abstract:This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. Drawing inspiration from seminal works in psycholinguistics, particularly the study of gender's influence on language perception, we leverage multilingual LLMs to revisit and expand upon the foundational experiments of Boroditsky (2003). Employing LLMs as a novel method for examining psycholinguistic biases related to grammatical gender, we prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. In particular, we look at adjective co-occurrences across gender and languages, and train a binary classifier to predict grammatical gender given adjectives an LLM uses to describe a noun. Surprisingly, we find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability. We show that while LLMs may describe words differently in different languages, they are biased similarly.

Computation and Language

What problem does this paper attempt to address?

The paper primarily explores whether large language models (LLMs) exhibit similar biases when handling different languages with grammatical gender. Specifically, the authors investigate the psycholinguistic biases of LLMs in different languages by having them describe nouns with specific genders using adjectives and analyzing whether these adjectives can predict the gender of the nouns. The main findings of the paper include: 1. **Bias in LLMs' description of nouns**: The authors found that even the same noun is described by different adjectives in different languages, but these adjectives can predict the grammatical gender of the noun. 2. **Cross-linguistic bias consistency**: Although LLMs may use different adjectives to describe nouns of the same gender in different languages, the biases among them are predictable, meaning that the biases exhibit consistency across different languages. 3. **Success of zero-shot transfer learning**: The authors also found that training a classifier to predict the gender of nouns in a specific language can be successfully applied to other unseen languages. This implies that even though LLMs may think differently in different languages, their biases related to grammatical gender are similar. In summary, this paper attempts to address whether large language models exhibit consistent biases when handling multiple languages with grammatical gender and whether these biases can be predicted across languages.

What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages

Assessing Gender Bias in LLMs: Comparing LLM Outputs with Human Perceptions and Official Statistics

Gender bias and stereotypes in Large Language Models

Gender Bias in Large Language Models across Multiple Languages

Evaluation of Large Language Models: STEM education and Gender Stereotypes

Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts

White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs

Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias

Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans

The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations

Probing Explicit and Implicit Gender Bias through LLM Conditional Text Generation

Investigating Markers and Drivers of Gender Bias in Machine Translations

Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?

Evaluating Gender Bias of LLMs in Making Morality Judgements

Evaluating Gender, Racial, and Age Biases in Large Language Models: A Comparative Analysis of Occupational and Crime Scenarios

Profiling Bias in LLMs: Stereotype Dimensions in Contextual Word Embeddings

Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora

Gender Bias of LLM in Economics: An Existentialism Perspective

Gender Bias in LLM-generated Interview Responses