Abstract:With the starting point that implicit human biases are reflected in the statistical regularities of language, it is possible to measure biases in English static word embeddings. State-of-the-art neural language models generate dynamic word embeddings dependent on the context in which the word appears. Current methods measure pre-defined social and intersectional biases that appear in particular contexts defined by sentence templates. Dispensing with templates, we introduce the Contextualized Embedding Association Test (CEAT), that can summarize the magnitude of overall bias in neural language models by incorporating a random-effects model. Experiments on social and intersectional biases show that CEAT finds evidence of all tested biases and provides comprehensive information on the variance of effect magnitudes of the same bias in different contexts. All the models trained on English corpora that we study contain biased representations. Furthermore, we develop two methods, Intersectional Bias Detection (IBD) and Emergent Intersectional Bias Detection (EIBD), to automatically identify the intersectional biases and emergent intersectional biases from static word embeddings in addition to measuring them in contextualized word embeddings. We present the first algorithmic bias detection findings on how intersectional group members are strongly associated with unique emergent biases that do not overlap with the biases of their constituent minority identities. IBD and EIBD achieve high accuracy when detecting the intersectional and emergent biases of African American females and Mexican American females. Our results indicate that biases at the intersection of race and gender associated with members of multiple minority groups, such as African American females and Mexican American females, have the highest magnitude across all neural language models.

Gender Bias Hidden Behind Chinese Word Embeddings: The Case of Chinese Adjectives

Analysis of Gender Bias in Social Perception and Judgement Using Chinese Word Embeddings

Examining Gender Bias in Languages with Grammatical Gender

Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics

Gender Bias in Contextualized Word Embeddings

Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer

Mitigating Gender Bias in Contextual Word Embeddings

Creating a Chinese gender lexicon for detecting gendered wording in job advertisements

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs

Measuring Gender Bias in Word Embeddings of Gendered Languages Requires Disentangling Grammatical Gender Signals

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words

Unveiling Gender Bias in Large Language Models: Using Teacher's Evaluation in Higher Education As an Example

Gender Bias in Large Language Models across Multiple Languages

Gender Bias in Neural Natural Language Processing

Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases

Social Biases in Automatic Evaluation Metrics for NLG

Exploration, detection, and mitigation: Unveiling gender bias in NLP

Unmasking the Stereotypes: Evaluating Social Biases in Chinese BERT

Attenuating Bias in Word Vectors

Assessing the Reliability of Word Embedding Gender Bias Measures