AI & robotics briefing: LLMs harbour hidden racism

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-00856-5

IF: 64.8

2024-03-21

Nature

Abstract:Chatbots make racist judgements on the basis of dialect. Plus, cyborg jellyfish could monitor the changing seas and how AI designed a drug. Chatbots make racist judgements on the basis of dialect. Plus, cyborg jellyfish could monitor the changing seas and how AI designed a drug.

multidisciplinary sciences

What problem does this paper attempt to address?

Chatbot AI makes racist judgements on the basis of dialect

Elizabeth Gibney

DOI: https://doi.org/10.1038/d41586-024-00779-1

IF: 64.8

2024-03-14

Nature

Abstract:Some large language models harbour hidden biases that cannot be removed using standard methods. Some large language models harbour hidden biases that cannot be removed using standard methods.

multidisciplinary sciences
AI & robotics briefing: Lack of transparency surrounds Neuralink's 'brain-reading' chip

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-00375-3

IF: 64.8

2024-02-08

Nature

Abstract:What scientists think of brain-computer interface company Neuralink's first human trial. Plus, an AI system learnt language by seeing the world through a baby's eyes and how ChatGPT could sway climate-change deniers. What scientists think of brain-computer interface company Neuralink's first human trial. Plus, an AI system learnt language by seeing the world through a baby's eyes and how ChatGPT could sway climate-change deniers.

multidisciplinary sciences
AI & robotics briefing: How psychology and neuroscience crack open the AI 'black box'

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-01538-y

IF: 64.8

2024-05-23

Nature

Abstract:Researchers are borrowing approaches from psychology and neuroscience to find out how large language models 'think'. Plus, self-driving super lab finds record-breaking laser material and AI-designed drugs get a reality check. Researchers are borrowing approaches from psychology and neuroscience to find out how large language models 'think'. Plus, self-driving super lab finds record-breaking laser material and AI-designed drugs get a reality check.

multidisciplinary sciences
AI & robotics briefing: What is AI, really?

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-02354-0

IF: 64.8

2024-07-18

Nature

Abstract:Complex maths or almost magic: scholars are divided on what AI really is and what it is capable of. Plus, AI analysis helps to unravel sperm whales' phonetic alphabet and algorithm predicts chemical composition of salt stains. Complex maths or almost magic: scholars are divided on what AI really is and what it is capable of. Plus, AI analysis helps to unravel sperm whales' phonetic alphabet and algorithm predicts chemical composition of salt stains.

multidisciplinary sciences
AI & robotics briefing: Can AI's bias problem be fixed?

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-00947-3

IF: 64.8

2024-03-28

Nature

Abstract:Researchers argue for more transparency to remove the biases that plague image generators. Plus, Nvidia's new 'superchip' promises AI performance and Google AI could soon use a person's cough to diagnose disease. Researchers argue for more transparency to remove the biases that plague image generators. Plus, Nvidia's new 'superchip' promises AI performance and Google AI could soon use a person's cough to diagnose disease.

multidisciplinary sciences
AI & robotics briefing: Why AI needs to see the 'ugly' side of science

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-01447-0

IF: 64.8

2024-05-15

Nature

Abstract:The absence of negative results in the scientific literature is affecting AI tools trained on published data. Plus, why animals still outrun robots and AlphaFold gets major upgrade. The absence of negative results in the scientific literature is affecting AI tools trained on published data. Plus, why animals still outrun robots and AlphaFold gets major upgrade.

multidisciplinary sciences
Fairness in AI-Driven Oncology: Investigating Racial and Gender Biases in Large Language Models

Anjali Agrawal

DOI: https://doi.org/10.7759/cureus.69541

2024-09-16

Cureus

Abstract:Introduction: Large language model (LLM) chatbots have many applications in medical settings. However, these tools can potentially perpetuate racial and gender biases through their responses, worsening disparities in healthcare. With the ongoing discussion of LLM chatbots in oncology and the widespread goal of addressing cancer disparities, this study focuses on biases propagated by LLM chatbots in oncology. Methods: Chat Generative Pre-trained Transformer (Chat GPT; OpenAI, San Francisco, CA, USA) was asked to determine what occupation a generic description of "assesses cancer patients" would correspond to for different demographics. Chat GPT, Gemini (Alphabet Inc., Mountain View, CA, USA), and Bing Chat (Microsoft Corp., Redmond, WA, USA) were prompted to provide oncologist recommendations in the top U.S. cities and demographic information (race, gender) of recommendations was compared against national distributions. Chat GPT was also asked to generate a job description for oncologists with different demographic backgrounds. Finally, Chat GPT, Gemini, and Bing Chat were asked to generate hypothetical cancer patients with race, smoking, and drinking histories. Results: LLM chatbots are about two times more likely to predict Blacks and Native Americans as oncology nurses than oncologists, compared to Asians (p < 0.01 and < 0.001, respectively). Similarly, they are also significantly more likely to predict females than males as oncology nurses (p < 0.001). Chat GPT's real-world oncologist recommendations overrepresent Asians by almost double and underrepresent Blacks by double and Hispanics by seven times. Chatbots also generate different job descriptions based on demographics, including cultural competency and advocacy and excluding treatment administration for underrepresented backgrounds. AI-generated cancer cases are not fully representative of real-world demographic distributions and encode stereotypes on substance abuse, such as Hispanics having a greater proportion of smokers than Whites by about 20% in Chat GPT breast cancer cases. Conclusion: To our knowledge, this is the first study of its kind to investigate racial and gender biases of such a diverse set of AI chatbots, and that too, within oncology. The methodology presented in this study provides a framework for targeted bias evaluation of LLMs in various fields across medicine.
LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions

Rumaisa Azeem,Andrew Hundt,Masoumeh Mansouri,Martim Brandão

2024-06-13

Abstract:Members of the Human-Robot Interaction (HRI) and Artificial Intelligence (AI) communities have proposed Large Language Models (LLMs) as a promising resource for robotics tasks such as natural language interactions, doing household and workplace tasks, approximating `common sense reasoning', and modeling humans. However, recent research has raised concerns about the potential for LLMs to produce discriminatory outcomes and unsafe behaviors in real-world robot experiments and applications. To address these concerns, we conduct an HRI-based evaluation of discrimination and safety criteria on several highly-rated LLMs. Our evaluation reveals that LLMs currently lack robustness when encountering people across a diverse range of protected identity characteristics (e.g., race, gender, disability status, nationality, religion, and their intersections), producing biased outputs consistent with directly discriminatory outcomes -- e.g. `gypsy' and `mute' people are labeled untrustworthy, but not `european' or `able-bodied' people. Furthermore, we test models in settings with unconstrained natural language (open vocabulary) inputs, and find they fail to act safely, generating responses that accept dangerous, violent, or unlawful instructions -- such as incident-causing misstatements, taking people's mobility aids, and sexual predation. Our results underscore the urgent need for systematic, routine, and comprehensive risk assessments and assurances to improve outcomes and ensure LLMs only operate on robots when it is safe, effective, and just to do so. Data and code will be made available.

Robotics,Artificial Intelligence,Computation and Language,Computers and Society
AI Language Models Could Both Help and Harm Equity in Marine Policymaking: The Case Study of the BBNJ Question-Answering Bot

Matt Ziegler,Sarah Lothian,Brian O'Neill,Richard Anderson,Yoshitaka Ota

2024-03-04

Abstract:AI Large Language Models (LLMs) like ChatGPT are set to reshape some aspects of policymaking processes. Policy practitioners are already using ChatGPT for help with a variety of tasks: from drafting statements, submissions, and presentations, to conducting background research. We are cautiously hopeful that LLMs could be used to promote a marginally more balanced footing among decision makers in policy negotiations by assisting with certain tedious work, particularly benefiting developing countries who face capacity constraints that put them at a disadvantage in negotiations. However, the risks are particularly concerning for environmental and marine policy uses, due to the urgency of crises like climate change, high uncertainty, and trans-boundary impact.

Computers and Society
AI generates covertly racist decisions about people based on their dialect

Valentin Hofmann,Pratyusha Ria Kalluri,Dan Jurafsky,Sharese King

DOI: https://doi.org/10.1038/s41586-024-07856-5

IF: 64.8

2024-08-31

Nature

Abstract:Hundreds of millions of people now interact with language models, with uses ranging from help with writing 1,2 to informing hiring decisions 3 . However, these language models are known to perpetuate systematic racial prejudices, making their judgements biased in problematic ways about groups such as African Americans 4,5,6,7 . Although previous research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time, particularly in the United States after the civil rights movement 8,9 . It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice, exhibiting raciolinguistic stereotypes about speakers of African American English (AAE) that are more negative than any human stereotypes about African Americans ever experimentally recorded. By contrast, the language models' overt stereotypes about African Americans are more positive. Dialect prejudice has the potential for harmful consequences: language models are more likely to suggest that speakers of AAE be assigned less-prestigious jobs, be convicted of crimes and be sentenced to death. Finally, we show that current practices of alleviating racial bias in language models, such as human preference alignment, exacerbate the discrepancy between covert and overt stereotypes, by superficially obscuring the racism that language models maintain on a deeper level. Our findings have far-reaching implications for the fair and safe use of language technology.

multidisciplinary sciences
Covert racism in AI chatbots, precise Stone Age engineering, and the science of paper cuts

Benjamin Thompson,Elizabeth Gibney,Emily Bates

DOI: https://doi.org/10.1038/d41586-024-02806-7

IF: 64.8

2024-08-31

Nature

Abstract:We round up some recent stories from the Nature Briefing. Hear the biggest stories from the world of science | 28 August 2024

multidisciplinary sciences
AI & robotics briefing: Tech giants are 'open-washing' their AI models

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-02122-0

IF: 64.8

2024-06-27

Nature

Abstract:Google, Microsoft and others are describing their AI models as open source while disclosing little key information. Plus, self-driving cars' safety record remains unclear and a new word for chatbot confabulations. Google, Microsoft and others are describing their AI models as open source while disclosing little key information. Plus, self-driving cars' safety record remains unclear and a new word for chatbot confabulations.

multidisciplinary sciences
Is ChatGPT More Biased Than You?

Golnoosh Babaei,David Banks,Costanza Bosone,Paolo Giudici,Yunhong Shan

DOI: https://doi.org/10.1162/99608f92.2781452d

2024-06-21

Harvard Data Science Review

Abstract:AI is changing the world in ways that are difficult to forecast, but the impact will surely be enormous. Large language models (LLMs) are the most recent AI system that has captured the public eye. The rise of AI and LLMs offers efficiency and assistance, but raises questions of job loss, fairness, and societal norms. Bias is a significant challenge. Educational reforms are needed, and legal frameworks must adapt to address liability and privacy issues. Ultimately, human choices will shape AI influence, highlighting the need for responsible development and regulation to ensure benefits outweigh risks.
Robots Enact Malignant Stereotypes

Andrew Hundt,William Agnew,Vicky Zeng,Severin Kacianka,Matthew Gombolay

DOI: https://doi.org/10.48550/arXiv.2207.11569

IF: 3.7

2022-07-23

Robotics

Abstract:Stereotypes, bias, and discrimination have been extensively documented in Machine Learning (ML) methods such as Computer Vision (CV) [18, 80], Natural Language Processing (NLP) [6], or both, in the case of large image and caption models such as OpenAI CLIP [14]. In this paper, we evaluate how ML bias manifests in robots that physically and autonomously act within the world. We audit one of several recently published CLIP-powered robotic manipulation methods, presenting it with objects that have pictures of human faces on the surface which vary across race and gender, alongside task descriptions that contain terms associated with common stereotypes. Our experiments definitively show robots acting out toxic stereotypes with respect to gender, race, and scientifically-discredited physiognomy, at scale. Furthermore, the audited methods are less likely to recognize Women and People of Color. Our interdisciplinary sociotechnical analysis synthesizes across fields and applications such as Science Technology and Society (STS), Critical Studies, History, Safety, Robotics, and AI. We find that robots powered by large datasets and Dissolution Models (sometimes called "foundation models", e.g. CLIP) that contain humans risk physically amplifying malignant stereotypes in general; and that merely correcting disparities will be insufficient for the complexity and scale of the problem. Instead, we recommend that robot learning methods that physically manifest stereotypes or other harmful outcomes be paused, reworked, or even wound down when appropriate, until outcomes can be proven safe, effective, and just. Finally, we discuss comprehensive policy changes and the potential of new interdisciplinary research on topics like Identity Safety Assessment Frameworks and Design Justice to better understand and address these harms.
Computer says 'no': Exploring systemic bias in ChatGPT using an audit approach

Louis Lippens

DOI: https://doi.org/10.1016/j.chbah.2024.100054

2024-02-12

Abstract:Large language models offer significant potential for increasing labour productivity, such as streamlining personnel selection, but raise concerns about perpetuating systemic biases embedded into their pre-training data. This study explores the potential ethnic and gender bias of ChatGPT, a chatbot producing human-like responses to language tasks, in assessing job applicants. Using the correspondence audit approach from the social sciences, I simulated a CV screening task with 34,560 vacancy-CV combinations where the chatbot had to rate fictitious applicant profiles. Comparing ChatGPT's ratings of Arab, Asian, Black American, Central African, Dutch, Eastern European, Hispanic, Turkish, and White American male and female applicants, I show that ethnic and gender identity influence the chatbot's evaluations. Ethnic discrimination is more pronounced than gender discrimination and mainly occurs in jobs with favourable labour conditions or requiring greater language proficiency. In contrast, gender discrimination emerges in gender-atypical roles. These findings suggest that ChatGPT's discriminatory output reflects a statistical mechanism echoing societal stereotypes. Policymakers and developers should address systemic bias in language model-driven applications to ensure equitable treatment across demographic groups. Practitioners should practice caution, given the adverse impact these tools can (re)produce, especially in selection decisions involving humans.

General Economics
Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing

J. Elliott Casal,Matthew Kessler,Matt Kessler

DOI: https://doi.org/10.1016/j.rmal.2023.100068

2023-12-01

Research Methods in Applied Linguistics

Abstract:There has been considerable intrigue surrounding the use of Large Language Model powered AI chatbots such as ChatGPT in research, educational contexts, and beyond. However, most studies have explored such tools’ general capabilities and applications for language teaching purposes. The current study advances this discussion to examine issues pertaining to human judgements, accuracy, and research ethics. Specifically, we investigate: 1) the extent to which linguists/reviewers from top journals can distinguish AI- from human-generated writing, 2) what the basis of reviewers’ decisions are, and 3) the extent to which editors of top Applied Linguistics journals believe AI tools are ethical for research purposes. In the study, reviewers (N = 72) completed a judgement task involving AI- and human-generated research abstracts, and several reviewers participated in follow-up interviews to explain their rationales. Similarly, editors (N = 27) completed a survey and interviews to discuss their beliefs. Findings suggest that despite employing multiple rationales to judge texts, reviewers were largely unsuccessful in identifying AI versus human writing, with an overall positive identification rate of only 38.9%. Additionally, many editors believed there are ethical uses of AI tools for facilitating research processes, yet some disagreed. Future research directions are discussed involving AI tools and academic publishing.

English Else
Bias of AI-generated content: an examination of news produced by large language models

Xiao Fang,Shangkun Che,Minjia Mao,Hongzhe Zhang,Ming Zhao,Xiaohang Zhao

DOI: https://doi.org/10.1038/s41598-024-55686-2

IF: 4.6

2024-03-04

Scientific Reports

Abstract:Large language models (LLMs) have the potential to transform our lives and work through the content they generate, known as AI-Generated Content (AIGC). To harness this transformation, we need to understand the limitations of LLMs. Here, we investigate the bias of AIGC produced by seven representative LLMs, including ChatGPT and LLaMA. We collect news articles from The New York Times and Reuters, both known for their dedication to provide unbiased news. We then apply each examined LLM to generate news content with headlines of these news articles as prompts, and evaluate the gender and racial biases of the AIGC produced by the LLM by comparing the AIGC and the original news articles. We further analyze the gender bias of each LLM under biased prompts by adding gender-biased messages to prompts constructed from these news headlines. Our study reveals that the AIGC produced by each examined LLM demonstrates substantial gender and racial biases. Moreover, the AIGC generated by each LLM exhibits notable discrimination against females and individuals of the Black race. Among the LLMs, the AIGC generated by ChatGPT demonstrates the lowest level of bias, and ChatGPT is the sole model capable of declining content generation when provided with biased prompts.

multidisciplinary sciences
Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption

Alaina N. Talboy,Elizabeth Fuller

2023-08-29

Abstract:Assessments of algorithmic bias in large language models (LLMs) are generally catered to uncovering systemic discrimination based on protected characteristics such as sex and ethnicity. However, there are over 180 documented cognitive biases that pervade human reasoning and decision making that are routinely ignored when discussing the ethical complexities of AI. We demonstrate the presence of these cognitive biases in LLMs and discuss the implications of using biased reasoning under the guise of expertise. We call for stronger education, risk management, and continued research as widespread adoption of this technology increases. Finally, we close with a set of best practices for when and how to employ this technology as widespread adoption continues to grow.

Human-Computer Interaction
AI & robotics briefing: 'Risk-of-death' AI save lives in hospital trial

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-01382-0

IF: 64.8

2024-05-09

Nature

Abstract:An algorithm trained on electrocardiograms reduced deaths among high-risk patients by 31%. Plus, an AI transcription tool seems to hallucinate harmful text and a robot that efficiently combines bouncing and flying. An algorithm trained on electrocardiograms reduced deaths among high-risk patients by 31%. Plus, an AI transcription tool seems to hallucinate harmful text and a robot that efficiently combines bouncing and flying.

multidisciplinary sciences
AI & robotics briefing: Machine vision won't take our jobs yet

Katrina Krämer

DOI: https://doi.org/10.1038/d41586-024-00284-5

IF: 64.8

2024-02-01

Nature

Abstract:Automating visual tasks using computer vision probably won't be worth it for many companies. Plus, two-faced AI can learn to hide deception and butterfly effect found in ChatGPT prompts. Automating visual tasks using computer vision probably won't be worth it for many companies. Plus, two-faced AI can learn to hide deception and butterfly effect found in ChatGPT prompts.

multidisciplinary sciences

AI & robotics briefing: LLMs harbour hidden racism

Chatbot AI makes racist judgements on the basis of dialect

AI & robotics briefing: Lack of transparency surrounds Neuralink's 'brain-reading' chip

AI & robotics briefing: How psychology and neuroscience crack open the AI 'black box'

AI & robotics briefing: What is AI, really?

AI & robotics briefing: Can AI's bias problem be fixed?

AI & robotics briefing: Why AI needs to see the 'ugly' side of science

Fairness in AI-Driven Oncology: Investigating Racial and Gender Biases in Large Language Models

LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actions

AI Language Models Could Both Help and Harm Equity in Marine Policymaking: The Case Study of the BBNJ Question-Answering Bot

AI generates covertly racist decisions about people based on their dialect

Covert racism in AI chatbots, precise Stone Age engineering, and the science of paper cuts

AI & robotics briefing: Tech giants are 'open-washing' their AI models

Is ChatGPT More Biased Than You?

Robots Enact Malignant Stereotypes

Computer says 'no': Exploring systemic bias in ChatGPT using an audit approach

Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing

Bias of AI-generated content: an examination of news produced by large language models

Challenging the appearance of machine intelligence: Cognitive bias in LLMs and Best Practices for Adoption

AI & robotics briefing: 'Risk-of-death' AI save lives in hospital trial

AI & robotics briefing: Machine vision won't take our jobs yet