Abstract:Hundreds of millions of people now interact with language models, with uses ranging from serving as a writing aid to informing hiring decisions. Yet these language models are known to perpetuate systematic racial prejudices, making their judgments biased in problematic ways about groups like African Americans. While prior research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time. It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice: we extend research showing that Americans hold raciolinguistic stereotypes about speakers of African American English and find that language models have the same prejudice, exhibiting covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement. By contrast, the language models' overt stereotypes about African Americans are much more positive. We demonstrate that dialect prejudice has the potential for harmful consequences by asking language models to make hypothetical decisions about people, based only on how they speak. Language models are more likely to suggest that speakers of African American English be assigned less prestigious jobs, be convicted of crimes, and be sentenced to death. Finally, we show that existing methods for alleviating racial bias in language models such as human feedback training do not mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level. Our findings have far-reaching implications for the fair and safe employment of language technology.

LLMs produce racist output when prompted in African American English

AI generates covertly racist decisions about people based on their dialect

Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans

Dialect prejudice predicts AI decisions about people's character, employability, and criminality

$\textit{Who Speaks Matters}$: Analysing the Influence of the Speaker's Ethnicity on Hate Classification

Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?

"Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models

LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users

The African Woman is Rhythmic and Soulful: An Investigation of Implicit Biases in LLM Open-ended Text Generation

Protected group bias and stereotypes in Large Language Models

Aligning with Whom? Large Language Models Have Gender and Racial Biases in Subjective NLP Tasks

Large language models propagate race-based medicine

LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Education

White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs

The Life Cycle of Large Language Models: A Review of Biases in Education

Gender bias and stereotypes in Large Language Models

Understanding Intrinsic Socioeconomic Biases in Large Language Models

Popular LLMs Amplify Race and Gender Disparities in Human Mobility

Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness