Abstract:Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks. These biases stem not only from language but also from the cultural knowledge required to interpret questions, reducing the practical utility of translated datasets like MMLU. Furthermore, translation often introduces artifacts that can distort the meaning or clarity of questions in the target language. A common practice in multilingual evaluation is to rely on machine-translated evaluation sets, but simply translating a dataset is insufficient to address these challenges. In this work, we trace the impact of both of these issues on multilingual evaluations and ensuing model performances. Our large-scale evaluation of state-of-the-art open and proprietary models illustrates that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge. Moreover, for questions requiring geographic knowledge, an astounding 84.9% focus on either North American or European regions. Rankings of model evaluations change depending on whether they are evaluated on the full portion or the subset of questions annotated as culturally sensitive, showing the distortion to model rankings when blindly relying on translated MMLU. We release Global-MMLU, an improved MMLU with evaluation coverage across 42 languages -- with improved overall quality by engaging with compensated professional and community annotators to verify translation quality while also rigorously evaluating cultural biases present in the original dataset. This comprehensive Global-MMLU set also includes designated subsets labeled as culturally sensitive and culturally agnostic to allow for more holistic, complete evaluation.

Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation

TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish

Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking

A Comprehensive Evaluation of Large Language Models for Turkish Abstractive Dialogue Summarization

ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic

Measuring Taiwanese Mandarin Language Understanding

Do LLMs Recognize me, When I is not me: Assessment of LLMs Understanding of Turkish Indexical Pronouns in Indexical Shift Contexts

Büyük dil modellerinin Türkçe verisetleri ile eğitilmesi ve ince ayarlanması

A Benchmark Evaluation of Multilingual Large Language Models for Arabic Cross-Lingual Named-Entity Recognition

Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training

MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

CMMLU: Measuring massive multitask language understanding in Chinese

A bilingual benchmark for evaluating large language models

Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (Published at NeurIPS 2024 Track Datasets and Benchmarks)

What do Large Language Models Need for Machine Translation Evaluation?

P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs

Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

KMMLU: Measuring Massive Multitask Language Understanding in Korean

Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models