Abstract:The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain

Conceptualized Representation Learning for Chinese Biomedical Text Mining

CLUE: A Chinese Language Understanding Evaluation Benchmark

MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models

Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark

CMB: A Comprehensive Medical Benchmark in Chinese

CHBench: A Chinese Dataset for Evaluating Health in Large Language Models

BioSentEval: An Evaluation Toolkit for Chinese Medical Sentence Representation.

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

Advanced PromptCBLUE Performance: A Novel Approach Leveraging Large Language Models

Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT

Overview of the PromptCBLUE Shared Task in CHIP2023

ECNU-LLM@CHIP-PromptCBLUE: Prompt Optimization and In-Context Learning for Chinese Medical Tasks

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

An Extensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPT

CHiSafetyBench: A Chinese Hierarchical Safety Benchmark for Large Language Models