Abstract:Background: Clinical notes contain contextualized information beyond structured data related to patients' past and current health status. Objective: This study aimed to design a multimodal deep learning approach to improve the evaluation precision of hospital outcomes for heart failure (HF) using admission clinical notes and easily collected tabular data. Methods: Data for the development and validation of the multimodal model were retrospectively derived from 3 open-access US databases, including the Medical Information Mart for Intensive Care III v1.4 (MIMIC-III) and MIMIC-IV v1.0, collected from a teaching hospital from 2001 to 2019, and the eICU Collaborative Research Database v1.2, collected from 208 hospitals from 2014 to 2015. The study cohorts consisted of all patients with critical HF. The clinical notes, including chief complaint, history of present illness, physical examination, medical history, and admission medication, as well as clinical variables recorded in electronic health records, were analyzed. We developed a deep learning mortality prediction model for in-hospital patients, which underwent complete internal, prospective, and external evaluation. The Integrated Gradients and SHapley Additive exPlanations (SHAP) methods were used to analyze the importance of risk factors. Results: The study included 9989 (16.4%) patients in the development set, 2497 (14.1%) patients in the internal validation set, 1896 (18.3%) in the prospective validation set, and 7432 (15%) patients in the external validation set. The area under the receiver operating characteristic curve of the models was 0.838 (95% CI 0.827-0.851), 0.849 (95% CI 0.841-0.856), and 0.767 (95% CI 0.762-0.772), for the internal, prospective, and external validation sets, respectively. The area under the receiver operating characteristic curve of the multimodal model outperformed that of the unimodal models in all test sets, and tabular data contributed to higher discrimination. The medical history and physical examination were more useful than other factors in early assessments. Conclusions: The multimodal deep learning model for combining admission notes and clinical tabular data showed promising efficacy as a potentially novel method in evaluating the risk of mortality in patients with HF, providing more accurate and timely decision support.

M3T-LM: A multi-modal multi-task learning model for jointly predicting patient length of stay and mortality

Multi-modal learning for inpatient length of stay prediction

Multimodal temporal-clinical note network for mortality prediction

Enhancing In-Hospital Mortality Prediction Using Multi-Representational Learning with LLM-Generated Expert Summaries

M3H: Multimodal Multitask Machine Learning for Healthcare

Integrating multi-task and cost-sensitive learning for predicting mortality risk of chronic diseases in the elderly using real-world data

M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities

DeepMPM: a Mortality Risk Prediction Model Using Longitudinal EHR Data.

M3Care: Learning with Missing Modalities in Multimodal Healthcare Data

Interpretive Clinical Measurements Prediction and Diagnosis in ICU with Multi-Task Attention-Guided Sequential Feature Learning

Deep Multi-path Network Integrating Incomplete Biomarker and Chest CT Data for Evaluating Lung Cancer Risk

Improving the Prognostic Evaluation Precision of Hospital Outcomes for Heart Failure Using Admission Notes and Clinical Tabular Data: Multimodal Deep Learning Model

Multimodal risk prediction with physiological signals, medical images and clinical notes

Predicting Length of Stay in ICU and Mortality with Temporal Dilated Separable Convolution and Context-Aware Feature Fusion.

DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis

M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

XAI for In-hospital Mortality Prediction via Multimodal ICU Data

Dual-Core Mutual Learning Between Scoring Systems and Clinical Features for ICU Mortality Prediction

DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data.

Interpretable Machine Learning Model for Early Prediction of Mortality in Elderly Patients with Multiple Organ Dysfunction Syndrome (MODS): a Multicenter Retrospective Study and Cross Validation

Multi-task oriented diffusion model for mortality prediction in shock patients with incomplete data