A Dataset and Benchmark for Hospital Course Summarization with Adapted Large Language Models

Asad Aali,Dave Van Veen,Yamin Ishraq Arefeen,Jason Hom,Christian Bluethgen,Eduardo Pontes Reis,Sergios Gatidis,Namuun Clifford,Joseph Daws,Arash S. Tehrani,Jangwon Kim,Akshay S. Chaudhari
2024-08-27
Abstract:Brief hospital course (BHC) summaries are clinical documents that summarize a patient's hospital stay. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as synthesizing BHCs from clinical notes have not been shown. We introduce a novel pre-processed dataset, the MIMIC-IV-BHC, encapsulating clinical note and brief hospital course (BHC) pairs to adapt LLMs for BHC synthesis. Furthermore, we introduce a benchmark of the summarization performance of two general-purpose LLMs and three healthcare-adapted LLMs. Using clinical notes as input, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to three open-source LLMs (Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5, GPT-4). We evaluate these LLMs across multiple context-length inputs using natural language similarity metrics. We further conduct a clinical study with five clinicians, comparing clinician-written and LLM-generated BHCs across 30 samples, focusing on their potential to enhance clinical decision-making through improved summary quality. We observe that the Llama2-13B fine-tuned LLM outperforms other domain-adapted models given quantitative evaluation metrics of BLEU and BERT-Score. GPT-4 with in-context learning shows more robustness to increasing context lengths of clinical note inputs than fine-tuned Llama2-13B. Despite comparable quantitative metrics, the reader study depicts a significant preference for summaries generated by GPT-4 with in-context learning compared to both Llama2-13B fine-tuned summaries and the original summaries, highlighting the need for qualitative clinical evaluation.
Computation and Language,Artificial Intelligence,Machine Learning
What problem does this paper attempt to address?
The paper aims to address the following issues: 1. **Automated Generation of Brief Hospital Course (BHC) Summaries**: Currently, clinicians need to spend a significant amount of time manually writing the brief hospital course summaries in discharge notes, which is not only time-consuming but also prone to errors. The goal of the paper is to utilize large language models (LLMs) to automatically generate high-quality BHC summaries, thereby reducing the workload of clinicians and improving the quality of the summaries. 2. **Evaluating the Performance of Different Language Models on the BHC Generation Task**: The paper introduces a new preprocessed dataset, MIMIC-IV-BHC, and evaluates the performance of various general-purpose and medical-adapted LLMs on the BHC generation task based on this dataset. Specifically, it includes two proprietary models (GPT-3.5 and GPT-4) and three open-source models (Clinical-T5-Large, Llama2-13B, and FLAN-UL2). 3. **Exploring the Effectiveness of Different Adaptation Strategies**: The study employs various adaptation strategies such as Prompting, In-Context Learning (ICL), and Fine-tuning to determine the best combination of model and adaptation method. 4. **Combining Quantitative and Qualitative Evaluations**: In addition to using common natural language processing metrics like BLEU, ROUGE-L, and BERT-Score for quantitative evaluation, a clinical reader study was conducted. Five expert-level clinicians were invited to score the generated summaries, ensuring that the evaluation results are both technically accurate and meet clinical practical needs. In summary, this research aims to develop and test a method that can automatically and efficiently generate high-quality BHC summaries, thereby alleviating the workload of clinicians and improving the quality of medical documentation.