CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models

Wenhong Zhu,Hongkun Hao,Zhiwei He,Yunze Song,Yumeng Zhang,Hanxu Hu,Yiran Wei,Rui Wang,Hongyuan Lu

2024-06-03

Abstract:We are currently in an era of fierce competition among various large language models (LLMs) continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination, and it wastes dozens of time and effort for researchers and engineers to download and try those contaminated models. To save our precious time, we propose a novel and useful method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs in a cleaner manner. Clean-Eval employs an LLM to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter the generated low-quality samples to narrow down this candidate set. The best candidate is finally selected from this set based on the BLEURT score. According to human assessment, this best candidate is semantically similar to the original contamination data but expressed differently. All candidates can form a new benchmark to evaluate the model. Our experiments illustrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.

Computation and Language

What problem does this paper attempt to address?

This paper focuses on the evaluation problem of large-scale language models (LLMs) under training data contamination. Since LLMs are often trained on data extracted from websites and public datasets, there may be overlap between the training data and evaluation benchmarks, resulting in data contamination and overestimation of model performance. The paper proposes a new method called Clean-Eval to address this issue. Clean-Eval utilizes a neural network model to rewrite and translate the contaminated data, generating a candidate set with the same meaning but different expressions. Then, low-quality samples are filtered out by a semantic detector, and the final evaluation set is selected based on BLEURT scores. Experiments show that Clean-Eval effectively restores the actual performance of contaminated models on different tasks, making evaluations more accurate. The paper also emphasizes the significance of clean evaluation in driving the development of the LLMs community.