Abstract:Importance: The use of large language models (LLMs) in medicine is increasing, with potential applications in electronic health records (EHR) to create patient cohorts or identify patients who meet clinical trial recruitment criteria. However, significant barriers remain, including the extensive computer resources required, lack of performance evaluation, and challenges in implementation. Objective: This study aims to propose and test a framework to detect disease diagnosis using a recent light LLM on French-language EHR documents. Specifically, it focuses on detecting gout ( in French), a ubiquitous French term that have multiple meanings beyond the disease. The study will compare the performance of the LLM-based framework with traditional natural language processing techniques and test its dependence on the parameter used. Design: The framework was developed using a training and testing set of 700 paragraphs assessing , issued from a random selection of retrospective EHR documents. All paragraphs were manually reviewed and classified by two health-care professionals (HCP) into disease (true gout) and non-disease (gold standard). The LLM's accuracy was tested using few-shot and chain-of-thought prompting and compared to a regular expression (regex)-based method, focusing on the effects of model parameters and prompt structure. The framework was further validated on 600 paragraphs assessing . Setting: The documents were sampled from the electronic health-records of a tertiary university hospital in Geneva, Switzerland. Participants: Adults over 18 years of age. Exposure: Meta's Llama 3 8B LLM or traditional method, against a gold standard. Main Outcomes and Measures: Positive and negative predictive value, as well as accuracy of tested models. Results: The LLM-based algorithm outperformed the regex method, achieving a 92.7% [88.7-95.4%] positive predictive value, a 96.6% [94.6-97.8%] negative predictive value, and an accuracy of 95.4% [93.6-96.7%] for gout. In the validation set on CPPD, accuracy was 94.1% [90.2-97.6%]. The LLM framework performed well over a wide range of parameter values. Conclusions and Relevance: LLMs were able to accurately detect disease diagnoses from EHRs, even in non-English languages. They could facilitate creating large disease registries in any language, improving disease care assessment and patient recruitment for clinical trials.

MedFrenchmark, a Small Set for Benchmarking Generative LLMs in Medical French

DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain

Multiple Choice Questions and Large Languages Models: A Case Study with Fictional Medical Data

CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

A Benchmark for Long-Form Medical Question Answering

MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering

Large Language Models in Healthcare: A Comprehensive Benchmark

MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models

MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

Towards Evaluating and Building Versatile Large Language Models for Medicine

Large language models encode clinical knowledge

JMedBench: A Benchmark for Evaluating Japanese Biomedical Large Language Models

Large Language Model Benchmarks in Medical Tasks

Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Evaluation of large language model performance on the Biomedical Language Understanding and Reasoning Benchmark

Large language models for accurate disease detection in electronic health records

Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models

Benchmarking Large Language Models in Evidence-Based Medicine

BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models

A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation