Abstract:Background: Large language models (LLMs) such as GPT-4 hold great promise as transformative tools in health care, ranging from automating administrative tasks to augmenting clinical decision making. However, these models also pose a danger of perpetuating biases and delivering incorrect medical diagnoses, which can have a direct, harmful impact on medical care. We aimed to assess whether GPT-4 encodes racial and gender biases that impact its use in health care. Methods: Using the Azure OpenAI application interface, this model evaluation study tested whether GPT-4 encodes racial and gender biases and examined the impact of such biases on four potential applications of LLMs in the clinical domain-namely, medical education, diagnostic reasoning, clinical plan generation, and subjective patient assessment. We conducted experiments with prompts designed to resemble typical use of GPT-4 within clinical and medical education applications. We used clinical vignettes from NEJM Healer and from published research on implicit bias in health care. GPT-4 estimates of the demographic distribution of medical conditions were compared with true US prevalence estimates. Differential diagnosis and treatment planning were evaluated across demographic groups using standard statistical tests for significance between groups. Findings: We found that GPT-4 did not appropriately model the demographic diversity of medical conditions, consistently producing clinical vignettes that stereotype demographic presentations. The differential diagnoses created by GPT-4 for standardised clinical vignettes were more likely to include diagnoses that stereotype certain races, ethnicities, and genders. Assessment and plans created by the model showed significant association between demographic attributes and recommendations for more expensive procedures as well as differences in patient perception. Interpretation: Our findings highlight the urgent need for comprehensive and transparent bias assessments of LLM tools such as GPT-4 for intended use cases before they are integrated into clinical care. We discuss the potential sources of these biases and potential mitigation strategies before clinical implementation. Funding: Priscilla Chan and Mark Zuckerberg.

Evaluating Anti-LGBTQIA+ Medical Bias in Large Language Models

Bias patterns in the application of LLMs for clinical decision support: A comprehensive study

Evaluation and mitigation of cognitive biases in medical language models

Addressing cognitive bias in medical language models

Evaluation of Bias Towards Medical Professionals in Large Language Models

Red Teaming Large Language Models in Medicine: Real-World Insights on Model Behavior

Large language models propagate race-based medicine

Evaluating and Addressing Demographic Disparities in Medical Large Language Models: A Systematic Review

How Can We Diagnose and Treat Bias in Large Language Models for Clinical Decision-Making?

Self-Diagnosis and Large Language Models: A New Front for Medical Misinformation

A toolbox for surfacing health equity harms and biases in large language models

Unveiling and Mitigating Bias in Mental Health Analysis with Large Language Models

DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using Large Language Models

Evaluating large language models on medical, lay language, and self-reported descriptions of genetic conditions

AffirmativeAI: Towards LGBTQ+ Friendly Audit Frameworks for Large Language Models

Mitigating the Risk of Health Inequity Exacerbated by Large Language Models

Detecting Bias and Enhancing Diagnostic Accuracy in Large Language Models for Healthcare

Can AI Relate: Testing Large Language Model Response for Mental Health Support

Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study

The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs)