How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies

Alina Leidinger,Richard Rogers

2024-08-01

Abstract:With the widespread availability of LLMs since the release of ChatGPT and increased public scrutiny, commercial model development appears to have focused their efforts on 'safety' training concerning legal liabilities at the expense of social impact evaluation. This mimics a similar trend which we could observe for search engine autocompletion some years prior. We draw on scholarship from NLP and search engine auditing and present a novel evaluation task in the style of autocompletion prompts to assess stereotyping in LLMs. We assess LLMs by using four metrics, namely refusal rates, toxicity, sentiment and regard, with and without safety system prompts. Our findings indicate an improvement to stereotyping outputs with the system prompt, but overall a lack of attention by LLMs under study to certain harms classified as toxic, particularly for prompts about peoples/ethnicities and sexual orientation. Mentions of intersectional identities trigger a disproportionate amount of stereotyping. Finally, we discuss the implications of these findings about stereotyping harms in light of the coming intermingling of LLMs and search and the choice of stereotyping mitigation policy to adopt. We address model builders, academics, NLP practitioners and policy makers, calling for accountability and awareness concerning stereotyping harms, be it for training data curation, leader board design and usage, or social impact measurement.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the issue of stereotypes and biases towards specific social groups in the text generated by large - language models (LLMs). Specifically, the paper focuses on the following points:

1. **To what extent do current safe - training practices address the harm of stereotypes?** Researchers explored whether the existing "safe training" effectively reduces the generation of stereotypes by evaluating the performance of different LLMs when dealing with stereotype - related prompts.

2. **Are there differences in the strictness of different LLMs in stereotype regulation?** Researchers compared the responses of several state - of - the - art LLMs when dealing with stereotypes to understand which models are more strict in avoiding generating stereotypes.

3. **How offensive or toxic is the content generated by LLMs for different social groups?** Researchers analyzed the stereotype generation situation of LLMs for different social groups (such as ethnicity, gender, sexual orientation, etc.), especially the toxicity level of these generated contents.

4. **Can adding safety - system prompts reduce stereotypes in LLMs' responses?** Researchers tested the changes in the content generated by LLMs after adding safety - system prompts to the input prompts to evaluate the effectiveness of this approach.

5. **Will changing the format (e.g., removing the chat template) bypass "safe" behavior?** Researchers also explored whether the responses of LLMs would become more toxic or stereotypical without using the chat template.

Through the exploration of these issues, the paper aims to provide suggestions for model developers, NLP practitioners, and policy - makers on how to better manage and mitigate the harm of stereotypes in LLMs.