Abstract:Mental health stigma prevents many individuals from receiving the appropriate care, and social psychology studies have shown that mental health tends to be overlooked in men. In this work, we investigate gendered mental health stigma in masked language models. In doing so, we operationalize mental health stigma by developing a framework grounded in psychology research: we use clinical psychology literature to curate prompts, then evaluate the models' propensity to generate gendered words. We find that masked language models capture societal stigma about gender in mental health: models are consistently more likely to predict female subjects than male in sentences about having a mental health condition (32% vs. 19%), and this disparity is exacerbated for sentences that indicate treatment-seeking behavior. Furthermore, we find that different models capture dimensions of stigma differently for men and women, associating stereotypes like anger, blame, and pity more with women with mental health conditions than with men. In showing the complex nuances of models' gendered mental health stigma, we demonstrate that context and overlapping dimensions of identity are important considerations when assessing computational models' social biases.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explore and analyze the issue of gendered mental health stigma in Masked Language Models (MLMs). Specifically, the authors seek to address this issue by investigating the following two main questions: 1. **Do MLMs associate mental health issues with a specific gender?** - The authors designed a series of prompt sentences reflecting different stages of healthcare (diagnosis, intention, and action) to explore whether MLMs are more likely to predict women as the subjects of mental health issues rather than men when generating sentences. The study found that MLMs indeed tend to associate mental health issues more closely with women, especially in sentences indicating help-seeking behavior, where this difference is more pronounced. 2. **How do MLMs' stereotypes of men and women with mental health issues differ?** - The authors created a set of prompt sentences describing stereotypes of people with mental health issues by rephrasing questions from the Attribution Questionnaire (AQ-27) used in psychological research. The study found that when generating sentences describing mental health issues, MLMs are more likely to associate stereotypes such as anger, blame, and sympathy with women, while associating avoidance and lack of help with men. ### Research Background and Methods - **Background**: Mental health stigma is a serious issue that hinders many people from seeking appropriate treatment. Social psychology research indicates that mental health issues manifest differently in men and women, with men's issues more likely to be overlooked and underestimated. - **Methods**: The authors developed a framework based on social psychology literature to assess gender bias in MLMs by designing specific prompt sentences. The study used multiple pre-trained language models, including RoBERTa, MentalRoBERTa, and ClinicalLongformer, and analyzed the models' biases by statistically evaluating the probabilities of generated words. ### Main Findings 1. **Gender Association**: - In all mental health-related prompt sentences, the RoBERTa model generated female subjects significantly more often than male subjects (32% vs. 19%). This gender difference was present across the diagnosis, intention, and action stages, and was particularly evident in sentences indicating help-seeking behavior. - The domain-specific pre-trained MentalRoBERTa model, while still tending to generate female subjects, showed a smaller gender difference (5% vs. 13%), indicating that such pre-training can reduce gender bias. 2. **Stereotypes**: - In terms of stereotypes describing mental health issues, all models tended to associate the dimensions of help and avoidance with men, meaning the public is less likely to help and more likely to avoid men with mental health issues. - The MentalRoBERTa model not only continued these biases from RoBERTa but also more frequently associated negative stereotypes such as anger, blame, and sympathy with women with mental health issues. ### Conclusion The authors made the following contributions in this study: 1. Introduced a framework based on psychological research to analyze gender bias in models within the context of mental health stigma. 2. Provided empirical results showing that MLMs indeed perpetuate societal neglect of men's mental health issues and associate certain negative stereotypes with specific genders. 3. Demonstrated through empirical studies of multiple models that training on domain-specific data can reduce bias in some aspects but may increase bias in others, revealing the complexity of measuring social biases. This study is significant for understanding the potential impact of NLP models in mental health applications and provides methodological references for future research.

Gendered Mental Health Stigma in Masked Language Models

Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks

Gender and Racial Fairness in Depression Research using Social Media

Psychotic White Men and Bipolar Black Women? Racialized and Gendered Implications of Mental Health Terminology

Gender and Cross-Cultural Differences in Social Media Disclosures of Mental Illness

Camouflaging, internalized stigma, and mental health in the general population

UnMASKed: Quantifying Gender Biases in Masked Language Models through Linguistically Informed Job Market Prompts

How people evaluate others with social anxiety disorder: A comparison to depression and general mental illness stigma.

Mental health stigma: Impact on mental health treatment attitudes and physical health

Artificial Intelligence in mental health and the biases of language based models

Are Models Biased on Text without Gender-related Language?

The Stigma of Psychiatric Disorders and the Gender, Ethnicity, and Education of the Perceiver

Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China

Fairness and bias correction in machine learning for depression prediction: results from four study populations

The Seeking Mental Health Care model: prediction of help-seeking for depressive symptoms by stigma and mental illness representations

Gender Bias When Using Artificial Intelligence to Assess Anorexia Nervosa on Social Media: Data-Driven Study

Identifying schizophrenia stigma on Twitter: a proof of principle model using service user supervised machine learning

Man up! The mental health‐feminine stereotype and its effect on the adoption of mental health apps

Gender bias in (non)-contextual clinical word embeddings for stereotypical medical categories

Understanding the public stigma of mental illness: a mixed-methods, multi-level, exploratory triangulation study

Unveiling and Mitigating Bias in Mental Health Analysis with Large Language Models