Abstract:The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

What problem does this paper attempt to address?

The paper aims to address the lack of evaluation standards for current Chinese medical large language models (LLMs). Specifically, the paper introduces MedBench, a large-scale Chinese medical benchmark dataset containing 40,041 questions from real exams and clinical cases. MedBench covers four main sections: the Chinese Medical Licensing Examination, the Standardized Residency Training Examination, the Attending Physician Qualification Examination, and real-world clinical cases. The paper points out that existing medical benchmark datasets have some limitations, such as data contamination issues and the inability to comprehensively assess medical knowledge and practical application skills. MedBench ensures the authenticity and authority of the data by collecting the latest validated exam questions and expert-annotated electronic health records. Additionally, the paper conducts extensive experiments and in-depth analysis, revealing the following key findings: 1. **Poor performance of Chinese medical LLMs**: Chinese medical LLMs perform poorly on MedBench, especially in terms of clinical knowledge and diagnostic accuracy, which require significant improvement. 2. **General domain LLMs possess some medical knowledge**: Some general domain LLMs demonstrate considerable medical knowledge, showing potential application value. 3. **Human evaluation results**: Human evaluation shows that ChatGPT has extensive knowledge in clinical practice, while current Chinese medical LLMs lack high-quality conversational abilities and sufficient medical knowledge. 4. **Insufficient case analysis capability**: ChatGPT performs poorly on case analysis questions that require deep medical understanding and clinical reasoning skills, indicating significant room for improvement in the medical field. In summary, this study proposes a comprehensive benchmark dataset that aligns with the actual medical conditions in mainland China and provides profound empirical results, revealing the capabilities and limitations of medical LLMs. Furthermore, the paper introduces Item Response Theory to further optimize the benchmark dataset, aiming to aid the development of the medical research community.

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models

CMB: A Comprehensive Medical Benchmark in Chinese

TCMBench: A Comprehensive Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine

Towards Evaluating and Building Versatile Large Language Models for Medicine

Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset

Large Language Model Benchmarks in Medical Tasks

MedCalc-Bench: Evaluating Large Language Models for Medical Calculations

Are Large Language Models True Healthcare Jacks-of-All-Trades? Benchmarking Across Health Professions Beyond Physician Exams

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Large Language Models in Healthcare: A Comprehensive Benchmark

CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making

PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking Large Language Models

JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models

Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Benchmarking Large Language Models in Evidence-Based Medicine

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models

A Benchmark for Long-Form Medical Question Answering