LLMs-based Few-Shot Disease Predictions using EHR: A Novel Approach Combining Predictive Agent Reasoning and Critical Agent Instruction

Hejie Cui,Zhuocheng Shen,Jieyu Zhang,Hui Shao,Lianhui Qin,Joyce C. Ho,Carl Yang
2024-03-20
Abstract:Electronic health records (EHRs) contain valuable patient data for health-related prediction tasks, such as disease prediction. Traditional approaches rely on supervised learning methods that require large labeled datasets, which can be expensive and challenging to obtain. In this study, we investigate the feasibility of applying Large Language Models (LLMs) to convert structured patient visit data (e.g., diagnoses, labs, prescriptions) into natural language narratives. We evaluate the zero-shot and few-shot performance of LLMs using various EHR-prediction-oriented prompting strategies. Furthermore, we propose a novel approach that utilizes LLM agents with different roles: a predictor agent that makes predictions and generates reasoning processes and a critic agent that analyzes incorrect predictions and provides guidance for improving the reasoning of the predictor agent. Our results demonstrate that with the proposed approach, LLMs can achieve decent few-shot performance compared to traditional supervised learning methods in EHR-based disease predictions, suggesting its potential for health-oriented applications.
Computation and Language,Artificial Intelligence,Machine Learning,Multiagent Systems
What problem does this paper attempt to address?
The paper aims to address the application of Electronic Health Records (EHR) data in disease prediction. Specifically, the paper explores the following points: 1. **Utilizing Large Language Models (LLMs) for Few-Shot Learning**: - Traditional methods rely on supervised learning, which requires large annotated datasets that are often difficult to obtain and costly. Therefore, researchers attempt to use large language models to convert structured patient visit data (such as diagnoses, lab test results, prescription information, etc.) into natural language narratives and evaluate the performance of these models in zero-shot and few-shot scenarios. 2. **Proposing a Novel Method Combining Predictor and Critic Agents**: - The paper proposes an innovative method that uses two different LLM agents: one is a predictor agent responsible for generating predictions and reasoning processes; the other is a critic agent responsible for analyzing incorrect predictions and providing improvement suggestions. In this way, the system can learn from mistakes and adapt to the specific challenges of EHR data. 3. **Experimental Validation and Performance Evaluation**: - Researchers conducted experimental validation on two datasets: the publicly available MIMIC-III dataset and the private CRADLE dataset. The experimental results show that the proposed EHR-CoAgent method outperforms traditional machine learning methods and other single-agent LLM methods on various evaluation metrics (including accuracy, sensitivity, specificity, and F1 score). Particularly in few-shot scenarios, EHR-CoAgent demonstrated significant advantages. In summary, the paper aims to achieve efficient disease prediction using EHR data by combining predictor and critic agents with large language models, and it has achieved good results in practical applications.