Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench

Yotam Perlitz,Ariel Gera,Ofir Arviv,Asaf Yehudai,Elron Bandel,Eyal Shnarch,Michal Shmueli-Scheuer,Leshem Choshen

2024-09-12

Abstract:Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is assessing the validity of the benchmarks themselves. This is most commonly done via Benchmark Agreement Testing (BAT), where new benchmarks are validated against established ones using some agreement metric (e.g., rank correlation). Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing. This deficiency can lead to invalid conclusions, fostering mistrust in benchmarks and upending the ability to properly choose the appropriate benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how some overlooked methodological choices can significantly influence BAT results, potentially undermining the validity of conclusions. To address these inconsistencies, we propose a set of best practices for BAT and demonstrate how utilizing these methodologies greatly improves BAT robustness and validity. To foster adoption and facilitate future research,, we introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers. Our findings underscore the necessity for standardized BAT, ensuring the robustness and validity of benchmark evaluations in the evolving landscape of language model research. BenchBench Package: <a class="link-external link-http" href="http://github.com/IBM/BenchBench" rel="external noopener nofollow">this http URL</a> Leaderboard: <a class="link-external link-http" href="http://hf.co/spaces/IBM/BenchBench" rel="external noopener nofollow">this http URL</a>

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the lack of consistency and standardized methods in the Benchmark Agreement Testing (BAT) process. Specifically, it points out that there are some overlooked methodological choices in current benchmark validation practices that can significantly impact BAT results, thereby questioning the validity of the conclusions. By analyzing over 50 well-known benchmarks, the authors demonstrate how certain overlooked methodological choices affect BAT results and propose a set of best practices to improve the consistency and validity of BAT.

The main objectives include:
1. **Revealing the impact of methodological differences**: Analyzing the impact of different methodological choices on BAT results, such as the selection of reference benchmarks, models involved in the comparison, and relevance metrics.
2. **Proposing standardized best practices**: Developing a set of standardized best practice guidelines to enhance the reliability and consistency of BAT results.
3. **Developing tools to facilitate practice**: Releasing a Python package named BenchBench to implement these best practices and providing a meta-benchmark to evaluate the performance of other benchmarks.
4. **Establishing a dynamic leaderboard**: Introducing the BenchBench leaderboard, allowing users to easily obtain comparison results between different benchmarks based on their required reference benchmarks.

Through these efforts, the paper hopes to improve the inconsistency and unreliability issues in the current BAT process, thereby enhancing researchers' confidence in new benchmarks and helping users choose appropriate benchmarks.