Abstract:Background: Artificial intelligence chatbots such as ChatGPT (OpenAI) have garnered excitement about their potential for delegating writing tasks ordinarily performed by humans. Many of these tasks (eg, writing recommendation letters) have social and professional ramifications, making the potential social biases in ChatGPT's underlying language model a serious concern. Objective: Three preregistered studies used the text analysis program Linguistic Inquiry and Word Count to investigate gender bias in recommendation letters written by ChatGPT in human-use sessions (N=1400 total letters). Methods: We conducted analyses using 22 existing Linguistic Inquiry and Word Count dictionaries, as well as 6 newly created dictionaries based on systematic reviews of gender bias in recommendation letters, to compare recommendation letters generated for the 200 most historically popular "male" and "female" names in the United States. Study 1 used 3 different letter-writing prompts intended to accentuate professional accomplishments associated with male stereotypes, female stereotypes, or neither. Study 2 examined whether lengthening each of the 3 prompts while holding the between-prompt word count constant modified the extent of bias. Study 3 examined the variability within letters generated for the same name and prompts. We hypothesized that when prompted with gender-stereotyped professional accomplishments, ChatGPT would evidence gender-based language differences replicating those found in systematic reviews of human-written recommendation letters (eg, more affiliative, social, and communal language for female names; more agentic and skill-based language for male names). Results: Significant differences in language between letters generated for female versus male names were observed across all prompts, including the prompt hypothesized to be neutral, and across nearly all language categories tested. Historically female names received significantly more social referents (5/6, 83% of prompts), communal or doubt-raising language (4/6, 67% of prompts), personal pronouns (4/6, 67% of prompts), and clout language (5/6, 83% of prompts). Contradicting the study hypotheses, some gender differences (eg, achievement language and agentic language) were significant in both the hypothesized and nonhypothesized directions, depending on the prompt. Heteroscedasticity between male and female names was observed in multiple linguistic categories, with greater variance for historically female names than for historically male names. Conclusions: ChatGPT reproduces many gender-based language biases that have been reliably identified in investigations of human-written reference letters, although these differences vary across prompts and language categories. Caution should be taken when using ChatGPT for tasks that have social consequences, such as reference letter writing. The methods developed in this study may be useful for ongoing bias testing among progressive generations of chatbots across a range of real-world scenarios. Trial Registration: OSF Registries osf.io/ztv96; https://osf.io/ztv96

Let’s Face It: We Are Biased, and It Should Not Be That Way

Gender Bias in English: In Search of Fair Language

Gender and cultural bias in student evaluations: Why representation matters

Approaches to address bias in letters of recommendation

Gender bias in reference letters for residency and academic medicine: a systematic review

Editor's Spotlight/Take 5: How Prominent Are Gender Bias, Racial Bias, and Score Inflation in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review

Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics

Assessing Gender Bias in Particle Physics and Social Science Recommendations for Academic Jobs

Attitudes about gender bias in language: A reevaluation

Gender Bias and Sexism in Language

Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting

Biased Humans, (Un)Biased Algorithms?

Inside the Black Box of Organizational Life: The Gendered Language of Performance Assessment

“Master” of none: Institutional language change linked to reduced gender bias.

“Broad” Impact: Perceptions of Sex/Gender-Related Psychology Journals

Gender stereotypes and inclusion in language teaching

Gender bias in teaching evaluations: the causal role of department gender composition

Gender bias in academia: A lifetime problem that needs solutions

Gender bias in resident evaluations: Natural language processing and competency evaluation

'Too male, too pale, too stale': a qualitative exploration of student experiences of gender bias within medical education

What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT