Abstract:This study introduces a novel multiagent ensemble method powered by LLMs to address a key challenge in ML - data labeling, particularly in large-scale EHR datasets. Manual labeling of such datasets requires domain expertise and is labor-intensive, time-consuming, expensive, and error-prone. To overcome this bottleneck, we developed an ensemble LLMs method and demonstrated its effectiveness in two real-world tasks: (1) labeling a large-scale unlabeled ECG dataset in MIMIC-IV; (2) identifying social determinants of health (SDOH) from the clinical notes of EHR. Trading off benefits and cost, we selected a pool of diverse open source LLMs with satisfactory performance. We treat each LLM's prediction as a vote and apply a mechanism of majority voting with minimal winning threshold for ensemble. We implemented an ensemble LLMs application for EHR data labeling tasks. By using the ensemble LLMs and natural language processing, we labeled MIMIC-IV ECG dataset of 623,566 ECG reports with an estimated accuracy of 98.2%. We applied the ensemble LLMs method to identify SDOH from social history sections of 1,405 EHR clinical notes, also achieving competitive performance. Our experiments show that the ensemble LLMs can outperform individual LLM even the best commercial one, and the method reduces hallucination errors. From the research, we found that (1) the ensemble LLMs method significantly reduces the time and effort required for labeling large-scale EHR data, automating the process with high accuracy and quality; (2) the method generalizes well to other text data labeling tasks, as shown by its application to SDOH identification; (3) the ensemble of a group of diverse LLMs can outperform or match the performance of the best individual LLM; and (4) the ensemble method substantially reduces hallucination errors. This approach provides a scalable and efficient solution to data-labeling challenges.

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Improving Clinical Expertise in Large Language Models Using Electronic Medical Records

EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling

EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation

Critical Care Studies Using Large Language Models Based on Electronic Healthcare Records: A Technical Note

medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs

Large Language Models and Medical Knowledge Grounding for Diagnosis Prediction

Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Multimodal risk prediction with physiological signals, medical images and clinical notes

Large language models enabled multiagent ensemble method for efficient EHR data labeling

Tool Calling: Enhancing Medication Consultation via Retrieval-Augmented Large Language Models

Leveraging A Medical Knowledge Graph into Large Language Models for Diagnosis Prediction

Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval

Multimodal Clinical Reasoning through Knowledge-augmented Rationale Generation

Retrieval-Augmented and Knowledge-Grounded Language Models for Faithful Clinical Medicine

EHR-KnowGen: Knowledge-enhanced multimodal learning for disease diagnosis generation

ColaCare: Enhancing Electronic Health Record Modeling through Large Language Model-Driven Multi-Agent Collaboration

Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook