Abstract:Large language models (LLMs) are transforming the ways the general public accesses and consumes information. Their influence is particularly pronounced in pivotal sectors like healthcare, where lay individuals are increasingly appropriating LLMs as conversational agents for everyday queries. While LLMs demonstrate impressive language understanding and generation proficiencies, concerns regarding their safety remain paramount in these high-stake domains. Moreover, the development of LLMs is disproportionately focused on English. It remains unclear how these LLMs perform in the context of non-English languages, a gap that is critical for ensuring equity in the real-world use of these systems.This paper provides a framework to investigate the effectiveness of LLMs as multi-lingual dialogue systems for healthcare queries. Our empirically-derived framework XlingEval focuses on three fundamental criteria for evaluating LLM responses to naturalistic human-authored health-related questions: correctness, consistency, and verifiability. Through extensive experiments on four major global languages, including English, Spanish, Chinese, and Hindi, spanning three expert-annotated large health Q&A datasets, and through an amalgamation of algorithmic and human-evaluation strategies, we found a pronounced disparity in LLM responses across these languages, indicating a need for enhanced cross-lingual capabilities. We further propose XlingHealth, a cross-lingual benchmark for examining the multilingual capabilities of LLMs in the healthcare context. Our findings underscore the pressing need to bolster the cross-lingual capacities of these models, and to provide an equitable information ecosystem accessible to all.

Framing Electronic Medical Records As Polylingual Documents in Query Expansion.

An Electronic Medical Record System with Treatment Recommendations Based on Patient Similarity.

LI-EMRSQL: Linking Information Enhanced Text2SQL Parsing on Complex Electronic Medical Records

Semantic-Enhanced Query Expansion System for Retrieving Medical Image Notes

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

Improving Chinese Electronic Medical Record Retrieval by Field Weight Assignment, Negation Detection, and Re-ranking

Re-Structuring and Specific Similarity Computation of Electronic Medical Records

Leveraging Multi-granularity Heterogeneous Graph for Chinese Electronic Medical Records Summarization

Query-Focused EHR Summarization to Aid Imaging Diagnosis

Enhanced Electronic Health Records Text Summarization Using Large Language Models

Learning to Refine Expansion Terms for Biomedical Information Retrieval using Semantic Resources.

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

A query interface for clinical research with Chinese electronic health record using Natural Language Processing

Bilingual Term Alignment from Comparable Corpora in English Discharge Summary and Chinese Discharge Summary.

Named entity recognition in Chinese electronic medical records based on multi-feature integration

An Empirical Study on Query Expansion and Documentexpans Ion in Information Retrieval.

From electronic health records to terminology base: A novel knowledge base enrichment approach

A Fusion Framework to Extract Typical Treatment Patterns from Electronic Medical Records

Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries

SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmark

Relation Extraction Based on Fusion Dependency Parsing from Chinese EMRs