SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research

Liangtai Sun,Yang Han,Zihan Zhao,Da Ma,Zhennan Shen,Baocai Chen,Lu Chen,Kai Yu

2023-08-25

Abstract:Recently, there has been growing interest in using Large Language Models (LLMs) for scientific research. Numerous benchmarks have been proposed to evaluate the ability of LLMs for scientific research. However, current benchmarks are mostly based on pre-collected objective questions. This design suffers from data leakage problem and lacks the evaluation of subjective Q/A ability. In this paper, we propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. In particular, we design a "dynamic" subset based on scientific principles to prevent evaluation from potential data leakage. Both objective and subjective questions are included in SciEval. These characteristics make SciEval a more effective benchmark for scientific research ability evaluation of LLMs. Comprehensive experiments on most advanced LLMs show that, although GPT-4 achieves SOTA performance compared to other LLMs, there is still substantial room for improvement, especially for dynamic questions. The data and codes are now publicly available.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address several key issues in the evaluation of current large language models (LLMs) in the field of scientific research:

1. **Incomplete Evaluation System**: Existing benchmarks are primarily based on pre-collected objective questions, lacking the assessment of subjective question capabilities, which leads to an incomplete evaluation system.
2. **Data Leakage Risk**: Since LLMs are trained using a large amount of data, there is a risk of data leakage during the evaluation process.
3. **Limited Subject Coverage**: Current benchmarks are mostly confined to specific scientific fields, lacking a general scientific evaluation benchmark.

To address these issues, the authors propose SciEval, a comprehensive multidisciplinary evaluation benchmark covering approximately 18,000 challenging questions in the three fundamental scientific fields of chemistry, physics, and biology. SciEval systematically evaluates scientific capabilities across four dimensions: basic knowledge, knowledge application, scientific computation, and research ability. Additionally, SciEval introduces a dynamic data generation mechanism to prevent potential data leakage issues and ensure the fairness and credibility of the evaluation results. Experimental results show that although GPT-4 performs better than other models on static data, there is still room for improvement on dynamic data, especially in molecular knowledge and physical computation.