Abstract:Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis. Although impressive results have been achieved, we find that existing benchmarks do not reflect the complexity of real medical reports and specialized in-depth reasoning capabilities. In this work, we introduced RJUA-MedDQA, a comprehensive benchmark in the field of medical specialization, which poses several challenges: comprehensively interpreting imgage content across diverse challenging layouts, possessing numerical reasoning ability to identify abnormal indicators and demonstrating clinical reasoning ability to provide statements of disease diagnosis, status and advice based on medical contexts. We carefully design the data generation pipeline and proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed at restoring textual and tabular content in medical report images. This method substantially enhances annotation efficiency, doubling the productivity of each annotator, and yields a 26.8% improvement in accuracy. We conduct extensive evaluations, including few-shot assessments of 5 LMMs which are capable of solving Chinese medical QA tasks. To further investigate the limitations and potential of current LMMs, we conduct comparative experiments on a set of strong LLMs by using image-text generated by ESRA method. We report the performance of baselines and offer several observations: (1) The overall performance of existing LMMs is still limited; however LMMs more robust to low-quality and diverse-structured images compared to LLMs. (3) Reasoning across context and image content present significant challenges. We hope this benchmark helps the community make progress on these challenging tasks in multi-modal medical document understanding and facilitate its application in healthcare.

What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams

A Question-Answering System over Traditional Chinese Medicine

A Joint Model For Question-Answering Over Traditional Chinese Medicine

Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset

Huatuo-26M, a Large-scale Chinese Medical QA Dataset

XQA: A Cross-lingual Open-domain Question Answering Dataset

Medical Exam Question Answering with Large-scale Reading Comprehension

Answering medical questions in Chinese using automatically mined knowledge and deep neural networks: an end-to-end solution

Development of an Extractive Clinical Question Answering Dataset with Multi-Answer and Multi-Focus Questions

TCMD: A Traditional Chinese Medicine QA Dataset for Evaluating Large Language Models

A medical question answering system using large language models and knowledge graphs

A Dataset of Open-Domain Question Answering with Multiple-Span Answers

On the Generation of Medical Question-Answer Pairs

A Benchmark for Long-Form Medical Question Answering

Biomedical Question Answering: A Survey of Approaches and Challenges

MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning

PathVQA: 30000+ Questions for Medical Visual Question Answering

Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering

A Chinese Question Answering System in Medical Domain

emrQA: A Large Corpus for Question Answering on Electronic Medical Records