Abstract:Implicit gender bias in Large Language Models (LLMs) is a well-documented problem, and implications of gender introduced into automatic translations can perpetuate real-world biases. However, some LLMs use heuristics or post-processing to mask such bias, making investigation difficult. Here, we examine bias in LLMss via back-translation, using the DeepL translation API to investigate the bias evinced when repeatedly translating a set of 56 Software Engineering tasks used in a previous study. Each statement starts with 'she', and is translated first into a 'genderless' intermediate language then back into English; we then examine pronoun-choice in the back-translated texts. We expand prior research in the following ways: (1) by comparing results across five intermediate languages, namely Finnish, Indonesian, Estonian, Turkish and Hungarian; (2) by proposing a novel metric for assessing the variation in gender implied in the repeated translations, avoiding the over-interpretation of individual pronouns, apparent in earlier work; (3) by investigating sentence features that drive bias; (4) and by comparing results from three time-lapsed datasets to establish the reproducibility of the approach. We found that some languages display similar patterns of pronoun use, falling into three loose groups, but that patterns vary between groups; this underlines the need to work with multiple languages. We also identify the main verb appearing in a sentence as a likely significant driver of implied gender in the translations. Moreover, we see a good level of replicability in the results, and establish that our variation metric proves robust despite an obvious change in the behaviour of the DeepL translation API during the course of the study. These results show that the back-translation method can provide further insights into bias in language models.

Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation

Investigating Markers and Drivers of Gender Bias in Machine Translations

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

Measuring Gender Bias in West Slavic Language Models

Evaluating Gender Bias in Machine Translation

Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters

A Multilingual Perspective on Probing Gender Bias

Interpreting Gender Bias in Neural Machine Translation: Multilingual Architecture Matters

Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer

Gender Bias in Text: Labeled Datasets and Lexicons

Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias

What is Your Favorite Gender, MLM? Gender Bias Evaluation in Multilingual Masked Language Models

On Evaluating and Mitigating Gender Biases in Multilingual Settings

Gender Lost In Translation: How Bridging The Gap Between Languages Affects Gender Bias in Zero-Shot Multilingual Translation

The Birth of Bias: A case study on the evolution of gender bias in an English language model

Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectives

Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias

Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora

Reducing a Male Bias in Language? Establishing the Efficiency of Three Different Gender-Fair Language Strategies