NormAd: A Benchmark for Measuring the Cultural Adaptability of Large Language Models

Abhinav Rao,Akhila Yerukola,Vishwa Shah,Katharina Reinecke,Maarten Sap

2024-07-11

Abstract:The integration of large language models (LLMs) into various global cultures fundamentally presents a challenge: LLMs must navigate interactions, respect social norms, and avoid transgressing cultural boundaries. However, it is still unclear if LLMs can adapt their outputs to diverse cultural norms. Our study focuses on this aspect. We introduce NormAd, a novel dataset, which includes 2.6k stories that represent social and cultural norms from 75 countries, to assess the ability of LLMs to adapt to different granular levels of socio-cultural contexts such as the country of origin, its associated cultural values, and prevalent social norms. Our study reveals that LLMs struggle with cultural reasoning across all contextual granularities, showing stronger adaptability to English-centric cultures over those from the Global South. Even with explicit social norms, the top-performing model, Mistral-7b-Instruct, achieves only 81.8% accuracy, lagging behind the 95.6% achieved by humans. Evaluation on NormAd further reveals that LLMs struggle to adapt to stories involving gift-giving across cultures. Due to inherent agreement or sycophancy biases, LLMs find it considerably easier to assess the social acceptability of stories that adhere to norms than those that deviate. Our benchmark measures the cultural adaptability (or lack thereof) of LLMs, emphasizing the potential to make these technologies more equitable and useful for global audiences. We release the NormAd dataset and its associated code on GitHub.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to evaluate the adaptability of large - language models (LLMs) in different cultural contexts. Specifically, researchers are concerned about whether LLMs can adjust their outputs according to different sociocultural environments, respect various social norms, and avoid violating cultural boundaries. However, it is still unclear whether these models can adapt to diverse cultural norms. To this end, they introduce a new dataset, NORM AD, which contains 2.6 thousand stories from 75 countries and aims to evaluate the adaptability of LLMs to different sociocultural backgrounds, including multi - granular sociocultural situations such as national origin, related cultural values, and general social norms.

The research reveals that existing LLMs have difficulties in cultural reasoning across all situation granularities, showing stronger adaptability to English - centric cultures than to cultures in the Global South. Even when explicit social norms are provided, the best - performing model, Mistral - 7b - Instruct, only achieves an accuracy of 81.8%, far lower than the 95.6% of humans. In addition, LLMs perform poorly when dealing with stories involving gift - giving in different cultures. Due to inherent agreement or sycophancy biases, LLMs find it much easier to evaluate the social acceptability of stories that conform to norms than those that deviate from norms. Through this benchmark test, the research emphasizes the importance of improving the fairness and practicality of these technologies among global audiences.