Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora

Erik Derner,Sara Sansalvador de la Fuente,Yoan Gutiérrez,Paloma Moreda,Nuria Oliver

2024-11-22

Abstract:Gender bias in text corpora that are used for a variety of natural language processing (NLP) tasks, such as for training large language models (LLMs), can lead to the perpetuation and amplification of societal inequalities. This phenomenon is particularly pronounced in gendered languages like Spanish or French, where grammatical structures inherently encode gender, making the bias analysis more challenging. A first step in quantifying gender bias in text entails computing biases in gender representation, i.e., differences in the prevalence of words referring to males vs. females. Existing methods to measure gender representation bias in text corpora have mainly been proposed for English and do not generalize to gendered languages due to the intrinsic linguistic differences between English and gendered languages. This paper introduces a novel methodology that leverages the contextual understanding capabilities of LLMs to quantitatively measure gender representation bias in Spanish corpora. By utilizing LLMs to identify and classify gendered nouns and pronouns in relation to their reference to human entities, our approach provides a robust analysis of gender representation bias in gendered languages. We empirically validate our method on four widely-used benchmark datasets, uncovering significant gender prevalence disparities with a male-to-female ratio ranging from 4:1 to 6:1. These findings demonstrate the value of our methodology for bias quantification in gendered language corpora and suggest its application in NLP, contributing to the development of more equitable language technologies.

Computation and Language,Computers and Society

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the quantification of gender - representation bias in gendered language corpora. Specifically, the paper focuses on the gender - bias problem existing in the text corpora used in training large - language models (LLMs), which may lead to the continuation and amplification of social inequalities. This problem is particularly prominent in gendered languages such as Spanish or French, because the grammatical structures of these languages inherently encode gender information, making bias analysis more complex.

To address this challenge, the paper proposes a new methodology that utilizes the powerful context - understanding ability of large - language models to quantitatively measure gender - representation bias in Spanish - language text corpora. By using LLMs to identify and classify gender nouns and pronouns related to human entities, this method can provide a robust analysis of gender - representation bias. The paper, through empirical verification on four widely - used benchmark datasets, reveals a significant gender - ratio imbalance, with the male - to - female ratio ranging from 4:1 to 6:1. These findings not only demonstrate the value of the proposed methodology in quantifying bias in gendered - language corpora but also emphasize its application potential in the field of natural - language processing, which is helpful for developing more equitable language technologies.