Abstract:Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be immune to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-free versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 110B in size. LiveBench is difficult, with top models achieving below 65% accuracy. We release all questions, code, and model answers. Questions will be added and updated on a monthly basis, and we will release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

LiveBench: A Challenging, Contamination-Free LLM Benchmark

Don't Make Your LLM an Evaluation Benchmark Cheater

Benchmarking Benchmark Leakage in Large Language Models

Training on the Benchmark Is Not All You Need

Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation

Investigating Data Contamination in Modern Benchmarks for Large Language Models

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

Data Contamination Through the Lens of Time

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs

Position: LLM Unlearning Benchmarks are Weak Measures of Progress

Addressing Data Leakage in HumanEval Using Combinatorial Test Design

Benchmark Data Contamination of Large Language Models: A Survey

Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models

NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark

Benchmarking Foundation Models with Language-Model-as-an-Examiner

Contamination Report for Multilingual Benchmarks

ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain

CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models

LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs