Abstract:Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support, which has been demonstrated by their competitive performances in Medical QA. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations written by medical doctors which can be leveraged to establish various gold-based upper-bounds for comparison with LLMs performance. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs still has large room for improvement, especially for languages other than English. Furthermore, and despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. So far the benchmark is available in four languages, but we hope that this work may encourage further development to other languages.

HEAD-QA: A Healthcare Dataset for Complex Reasoning

Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering

RealMedQA: A pilot biomedical question answering dataset containing realistic clinical questions

Question Answering for Complex Electronic Health Records Database using Unified Encoder-Decoder Architecture

K-QA: A Real-World Medical Q&A Benchmark

EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images

MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers

Development of an Extractive Clinical Question Answering Dataset with Multi-Answer and Multi-Focus Questions

ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram

What Disease Does This Patient Have? A Large-Scale Open Domain Question Answering Dataset from Medical Exams

DSQA: A Domain Specific QA System for Smart Health Based on Knowledge Graph.

Knowledge Graph-based Question Answering with Electronic Health Records

A Joint-Reasoning based Disease Q&A System

CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data

MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering

Give me Some Hard Questions: Synthetic Data Generation for Clinical QA

BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English

EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering

Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks

Learning to Answer Multilingual and Code-Mixed Questions