[Pedigree of Waardenburg's syndrome in a family].

Y. Kawaguchi,Y. Kawa,S. Kawaguchi,M. Mizoguchi,N. Kubota

Abstract:

What problem does this paper attempt to address?

Poem: lichenplanus.

G. Savage

Western Journal of Medicine

Abstract:
Multi-modal learning for inpatient length of stay prediction

Junde Chen,Yuxin Wen,Michael Pokojovy,Tzu-Liang Tseng,Peter McCaffrey,Alexander Vo,Eric Walser,Scott Moen,Tzu-Liang (Bill) Tseng

DOI: https://doi.org/10.1016/j.compbiomed.2024.108121

IF: 7.7

2024-02-22

Computers in Biology and Medicine

Abstract:Predicting inpatient length of stay (LoS) is important for hospitals aiming to improve service efficiency and enhance management capabilities. Patient medical records are strongly associated with LoS. However, due to diverse modalities, heterogeneity, and complexity of data, it becomes challenging to effectively leverage these heterogeneous data to put forth a predictive model that can accurately predict LoS. To address the challenge, this study aims to establish a novel data-fusion model, termed as DF-Mdl, to integrate heterogeneous clinical data for predicting the LoS of inpatients between hospital discharge and admission. Multi-modal data such as demographic data, clinical notes, laboratory test results, and medical images are utilized in our proposed methodology with individual "basic" sub-models separately applied to each different data modality. Specifically, a convolutional neural network (CNN) model, which we termed CRXMDL, is designed for chest X-ray (CXR) image data, two long short-term memory networks are used to extract features from long text data, and a novel attention-embedded 1D convolutional neural network is developed to extract useful information from numerical data. Finally, these basic models are integrated to form a new data-fusion model (DF-Mdl) for inpatient LoS prediction. The proposed method attains the best R 2 and E VAR values of 0.6039 and 0.6042 among competitors for the LoS prediction on the Medical Information Mart for Intensive Care (MIMIC)-IV test dataset. Empirical evidence suggests better performance compared with other state-of-the-art (SOTA) methods, which demonstrates the effectiveness and feasibility of the proposed approach.

engineering, biomedical,computer science, interdisciplinary applications,mathematical & computational biology,biology
Dual-Core Mutual Learning Between Scoring Systems and Clinical Features for ICU Mortality Prediction

Zhenkun Shi,Sen Wang,Lin Yue,Yijia Zhang,Binod Kumar Adhikari,Shuai Xue,Wanli Zuo,Xue Li

DOI: https://doi.org/10.1016/j.ins.2023.118984

IF: 8.1

2023-04-22

Information Sciences

Abstract:Perpetually improving mortality prediction in intensive care units (ICUs) via the implementation of eHealth evaluation approaches has become a major research hotspot in the field of medical data mining for the purpose of saving lives. Recently, researchers have attempted to achieve improved prediction accuracy by using only deep learning-based techniques. However, some problems remain. (1) Most of the existing methods utilize independent clinical features to predict mortality by eliminating the correlations between the latent features, but this technique may fail to comprehensively capture and evaluate the statuses of patients. (2) Several clinical features are needed to ensure strong prediction accuracy, but most methods only use static features that are not extendable. (3) An effective practical framework that unifies traditional ICU scoring systems and state-of-the-art deep learning methods to predict mortality is also lacking. (4) Moreover, the interpretability of existing deep learning-based methods needs to be further improved. Therefore, we propose a novel d ual-core m utual l earning f ramework (DMLF) between ICU scoring systems and clinical features for mortality prediction. In particular, we mutually utilize s equential o rgan f ailure a ssessment (SOFA) scores and clinical measurement features to learn a unified model for enhancing the accuracy and interpretability of our DMLF. Experiments conducted on five real-world disease datasets show that the DMLF achieves significantly better prediction accuracy and area under the receiver operating characteristic curve (AUROC) values than six baselines and four state-of-the-art methods. Moreover, clinicians utilize a familiarized SOFA system to conduct mortality prediction and achieve increased interpretability, which benefits the adoption of the proposed framework in real clinical scenarios.

computer science, information systems
Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality Prediction

Satya Narayan Shukla,Benjamin M. Marlin

DOI: https://doi.org/10.48550/arXiv.2003.11059

2021-03-19

Abstract:Intensive Care Unit Electronic Health Records (ICU EHRs) store multimodal data about patients including clinical notes, sparse and irregularly sampled physiological time series, lab results, and more. To date, most methods designed to learn predictive models from ICU EHR data have focused on a single modality. In this paper, we leverage the recently proposed interpolation-prediction deep learning architecture(Shukla and Marlin 2019) as a basis for exploring how physiological time series data and clinical notes can be integrated into a unified mortality prediction model. We study both early and late fusion approaches and demonstrate how the relative predictive value of clinical text and physiological data change over time. Our results show that a late fusion approach can provide a statistically significant improvement in mortality prediction performance over using individual modalities in isolation.

Machine Learning,Computers and Society
Fusion of medical imaging and electronic health records with attention and multi-head machanisms

Cheng Jiang,Yihao Chen,Jianbo Chang,Ming Feng,Renzhi Wang,Jianhua Yao

DOI: https://doi.org/10.48550/arXiv.2112.11710

2021-12-22

Abstract:Doctors often make diagonostic decisions based on patient's image scans, such as magnetic resonance imaging (MRI), and patient's electronic health records (EHR) such as age, gender, blood pressure and so on. Despite a lot of automatic methods have been proposed for either image or text analysis in computer vision or natural language research areas, much fewer studies have been developed for the fusion of medical image and EHR data for medical problems. Among existing early or intermediate fusion methods, concatenation of features from both modalities is still a mainstream. For a better exploiting of image and EHR data, we propose a multi-modal attention module which use EHR data to help the selection of important regions during image feature extraction process conducted by traditional CNN. Moreover, we propose to incorporate multi-head machnism to gated multimodal unit (GMU) to make it able to parallelly fuse image and EHR features in different subspaces. With the help of the two modules, existing CNN architecture can be enhanced using both modalities. Experiments on predicting Glasgow outcome scale (GOS) of intracerebral hemorrhage patients and classifying Alzheimer's Disease showed the proposed method can automatically focus on task-related areas and achieve better results by making better use of image and EHR features.

Computer Vision and Pattern Recognition
Research on Multimodal Fusion of Temporal Electronic Medical Records

Moxuan Ma,Muyu Wang,Binyu Gao,Yichen Li,Jun Huang,Hui Chen

DOI: https://doi.org/10.3390/bioengineering11010094

2024-01-18

Abstract:The surge in deep learning-driven EMR research has centered on harnessing diverse data forms. Yet, the amalgamation of diverse modalities within time series data remains an underexplored realm. This study probes a multimodal fusion approach, merging temporal and non-temporal clinical notes along with tabular data. We leveraged data from 1271 myocardial infarction and 6450 stroke inpatients at a Beijing tertiary hospital. Our dataset encompassed static, and time series note data, coupled with static and time series table data. The temporal data underwent a preprocessing phase, padding to a 30-day interval, and segmenting into 3-day sub-sequences. These were fed into a long short-term memory (LSTM) network for sub-sequence representation. Multimodal attention gates were implemented for both static and temporal subsequence representations, culminating in fused representations. An attention-backtracking module was introduced for the latter, adept at capturing enduring dependencies in temporal fused representations. The concatenated results were channeled into an LSTM to yield the ultimate fused representation. Initially, two note modalities were designated as primary modes, and subsequently, the proposed fusion model was compared with comparative models including recent models such as Crossformer. The proposed model consistently exhibited superior predictive prowess in both tasks. Removing the attention-backtracking module led to performance decline. The proposed model consistently shows excellent predictive capabilities in both tasks. The proposed method not only effectively integrates data from the four modalities, but also has a good understanding of how to handle irregular time series data and lengthy clinical texts. An effective method is provided, which is expected to be more widely used in multimodal medical data representation.
Multi-channel fusion LSTM for medical event prediction using EHRs

Sicen Liu,Xiaolong Wang,Yang Xiang,Hui Xu,Hui Wang,Buzhou Tang

DOI: https://doi.org/10.1016/j.jbi.2022.104011

IF: 8

2022-03-01

Journal of Biomedical Informatics

Abstract:Automatic medical event prediction (MEP), e.g. diagnosis prediction, medication prediction, using electronic health records (EHRs) is a popular research direction in health informatics. In many cases, MEP relies on the determinations from different types of medical events, which demonstrates the heterogeneous nature of EHRs. However, most existing methods for MEP fail to distinguishingly model the type of event that is highly associated with the prediction task, i.e. task-wise event, which usually plays a more significant role than other events. In this paper, we proposed a Long Short-Term Memory network (LSTM)-based method for MEP, named Multi-Channel Fusion LSTM (MCF-LSTM), which models the correlations between different types of medical events using multiple network channels. To this end, we designed a task-wise fusion module, in which a gated network is applied to select how much information can be transferred between events. Furthermore, the irregular temporal interval between adjacent medical visits is also modeled in an individual channel, which is combined with other events in a unified manner. We compared MCF-LSTM with state-of-the-art methods on four MEP tasks on two public datasets: MIMIC-III and eICU. Experimental results show that MCF-LSTM achieves promising results on AUC(receiver operating characteristic curve), AUPR (area under the precision-recall curve), and top-k recall, and outperforms other methods with high stability.

medical informatics,computer science, interdisciplinary applications
DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data.

Yujuan Feng,Zhenxing Xu,Lin Gan,Ning Chen,Bin Yu,Ting Chen,Fei Wang

DOI: https://doi.org/10.1109/icdm.2019.00030

2019-01-01

Abstract:More and more healthcare data are becoming readily available nowadays. These data can help the healthcare professionals and patient themselves to better understand the patient status and potentially lead to improved care quality. However, the analysis of these data are challenging because they are large-scale and heterogeneous, high-dimensional and sparse, temporal but irregularly sampled. In this paper, we propose a method called Double Core Memory Networks (DCMN) to integrate information from different modalities of the longitudinal patient data and learn a joint patient representation effective for downstream analytical tasks such as risk prediction. DCMN is designed not only to disentangle the temporal and non-linear intra-modal dependencies for the data within each modality but also to capture the long-term inter-modal interactions. DCMN models are the end-to-end memory networks with two external memory cores where each modality of data is compressed and stored. Each memory core has an information-flow controller named query to interact with an external memory module. In addition, we incorporate a gating mechanism into basic DCMN model to perform dynamic regulation of memory interaction. DCMN models have multiple computational layers (hops) allowing data of different modalities interacting with each other recurrently along with a mechanism of alternating access of external memory for each memory core hop-by-hop. We evaluate DCMN models on two outcome prediction tasks, including a mortality prediction on the public Medical Information Mart for Intensive Care III (MIMIC-III) database and a cost prediction on the Hospital Quality Monitoring System (HQMS) dataset. Experimental results demonstrate that our DCMN models are more competitive over the baseline methods in the multimodal prediction setting.
Monitoring ICU Mortality Risk with A Long Short-Term Memory Recurrent Neural Network

Ke Yu,Mingda Zhang,Tianyi Cui,Milos Hauskrecht

Abstract:In intensive care units (ICU), mortality prediction is a critical factor not only for effective medical intervention but also for allocation of clinical resources. Structured electronic health records (EHR) contain valuable information for assessing mortality risk in ICU patients, but current mortality prediction models usually require laborious human-engineered features. Furthermore, substantial missing data in EHR is a common problem for both the construction and implementation of a prediction model.Inspired by language-related models, we design a new framework for dynamic monitoring of patients' mortality risk. Our framework uses the bag-of-words representation for all relevant medical events based on most recent history as inputs. By design, it is robust to missing data in EHR and can be easily implemented as an instant scoring system to monitor the medical development of all ICU patients. Specifically, our model uses latent semantic analysis (LSA) to encode the patients' states into low-dimensional embeddings, which are further fed to long short-term memory networks for mortality risk prediction. Our results show that the deep learning based framework performs better than the existing severity scoring system, SAPS-II. We observe that bidirectional long short-term memory demonstrates superior performance, probably due to the successful capture of both forward and backward temporal dependencies.
MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

Nasir Hayat,Krzysztof J. Geras,Farah E. Shamout

DOI: https://doi.org/10.48550/arXiv.2207.07027

2022-07-14

Image and Video Processing

Abstract:Multi-modal fusion approaches aim to integrate information from different data sources. Unlike natural datasets, such as in audio-visual applications, where samples consist of "paired" modalities, data in healthcare is often collected asynchronously. Hence, requiring the presence of all modalities for a given sample is not realistic for clinical tasks and significantly limits the size of the dataset during training. In this paper, we propose MedFuse, a conceptually simple yet promising LSTM-based fusion module that can accommodate uni-modal as well as multi-modal input. We evaluate the fusion method and introduce new benchmark results for in-hospital mortality prediction and phenotype classification, using clinical time-series data in the MIMIC-IV dataset and corresponding chest X-ray images in MIMIC-CXR. Compared to more complex multi-modal fusion strategies, MedFuse provides a performance improvement by a large margin on the fully paired test set. It also remains robust across the partially paired test set containing samples with missing chest X-ray images. We release our code for reproducibility and to enable the evaluation of competing models in the future.
Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Hejie Cui,Xinyu Fang,Ran Xu,Xuan Kan,Joyce C. Ho,Carl Yang

2024-02-20

Abstract:Electronic Health Records (EHRs) have become increasingly popular to support clinical decision-making and healthcare in recent decades. EHRs usually contain heterogeneous information, such as structural data in tabular form and unstructured data in textual notes. Different types of information in EHRs can complement each other and provide a more complete picture of the health status of a patient. While there has been a lot of research on representation learning of structured EHR data, the fusion of different types of EHR data (multimodal fusion) is not well studied. This is mostly because of the complex medical coding systems used and the noise and redundancy present in the written notes. In this work, we propose a new framework called MINGLE, which integrates both structures and semantics in EHR effectively. Our framework uses a two-level infusion strategy to combine medical concept semantics and clinical note semantics into hypergraph neural networks, which learn the complex interactions between different types of data to generate visit representations for downstream prediction. Experiment results on two EHR datasets, the public MIMIC-III and private CRADLE, show that MINGLE can effectively improve predictive performance by 11.83% relatively, enhancing semantic integration as well as multimodal fusion for structural and textual EHR data.

Machine Learning,Artificial Intelligence,Computation and Language
Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions

Suhan Cui,Jiaqi Wang,Yuan Zhong,Han Liu,Ting Wang,Fenglong Ma

2024-01-20

Abstract:The widespread adoption of Electronic Health Record (EHR) systems in healthcare institutes has generated vast amounts of medical data, offering significant opportunities for improving healthcare services through deep learning techniques. However, the complex and diverse modalities and feature structures in real-world EHR data pose great challenges for deep learning model design. To address the multi-modality challenge in EHR data, current approaches primarily rely on hand-crafted model architectures based on intuition and empirical experiences, leading to sub-optimal model architectures and limited performance. Therefore, to automate the process of model design for mining EHR data, we propose a novel neural architecture search (NAS) framework named AutoFM, which can automatically search for the optimal model architectures for encoding diverse input modalities and fusion strategies. We conduct thorough experiments on real-world multi-modal EHR data and prediction tasks, and the results demonstrate that our framework not only achieves significant performance improvement over existing state-of-the-art methods but also discovers meaningful network architectures effectively.

Machine Learning,Artificial Intelligence
Self-Supervised Predictive Coding with Multimodal Fusion for Patient Deterioration Prediction in Fine-grained Time Resolution

Kwanhyung Lee,John Won,Heejung Hyun,Sangchul Hahn,Edward Choi,Joohyung Lee

DOI: https://doi.org/10.48550/arXiv.2210.16598

2023-04-13

Abstract:Accurate time prediction of patients' critical events is crucial in urgent scenarios where timely decision-making is important. Though many studies have proposed automatic prediction methods using Electronic Health Records (EHR), their coarse-grained time resolutions limit their practical usage in urgent environments such as the emergency department (ED) and intensive care unit (ICU). Therefore, in this study, we propose an hourly prediction method based on self-supervised predictive coding and multi-modal fusion for two critical tasks: mortality and vasopressor need prediction. Through extensive experiments, we prove significant performance gains from both multi-modal fusion and self-supervised predictive regularization, most notably in far-future prediction, which becomes especially important in practice. Our uni-modal/bi-modal/bi-modal self-supervision scored 0.846/0.877/0.897 (0.824/0.855/0.886) and 0.817/0.820/0.858 (0.807/0.81/0.855) with mortality (far-future mortality) and with vasopressor need (far-future vasopressor need) prediction data in AUROC, respectively.

Machine Learning
Prediction of Mortality Events of Patients with Acute Heart Failure in Intensive Care Unit Based on Deep Neural Network

Jicheng Huang,Yufeng Cai,Xusheng Wu,Xin Huang,Jianwei Liu,Dehua Hu

DOI: https://doi.org/10.1016/j.cmpb.2024.108403

IF: 6.1

2024-09-03

Computer Methods and Programs in Biomedicine

Abstract:Background : Acute heart failure (AHF) in the intensive care unit (ICU) is characterized by its criticality, rapid progression, complex and changeable condition, and its pathophysiological process involves the interaction of multiple organs and systems. This makes it difficult to predict in-hospital mortality events comprehensively and accurately. Traditional analysis methods based on statistics and machine learning suffer from insufficient model performance, poor accuracy caused by prior dependence, and difficulty in adequately considering the complex relationships between multiple risk factors. Therefore, the application of deep neural network (DNN) techniques to the specific scenario, predicting mortality events of patients with AHF under intensive care, has become a research frontier. Methods : This research utilized the MIMIC-IV critical care database as the primary data source and employed the synthetic minority over-sampling technique (SMOTE) to balance the dataset. Deep neural network models—backpropagation neural network (BPNN) and recurrent neural network (RNN), which are based on electronic medical record data mining, were employed to investigate the in-hospital death event judgment task of patients with AHF under intensive care. Additionally, multiple single machine learning models and ensemble learning models were constructed for comparative experiments. Moreover, we achieved various optimal performance combinations by modifying the classification threshold of deep neural network models to address the diverse real-world requirements in the ICU. Finally, we conducted an interpretable deep model using SHapley Additive exPlanations (SHAP) to uncover the most influential medical record features for each patient from the aspects of global and local interpretation. Results : In terms of model performance in this scenario, deep neural network models outperform both single machine learning models and ensemble learning models, achieving the highest Accuracy, Precision, Recall, F1 value, and Area under the ROC curve, which can reach 0.949, 0.925, 0.983, 0.953, and 0.987 respectively. SHAP value analysis revealed that the ICU scores (APSIII, OASIS, SOFA) are significantly correlated with the occurrence of in-hospital fatal events. Conclusions : Our study underscores that DNN-based mortality event classifier offers a novel intelligent approach for forecasting and assessing the prognosis of AHF patients in the ICU. Additionally, the ICU scores stand out as the most predictive features, which implies that in the decision-making process of the models, ICU scores can provide the most crucial information, making the greatest positive or negative contribution to influence the incidence of in-hospital mortality among patients with acute heart failure.

engineering, biomedical,computer science, interdisciplinary applications,medical informatics, theory & methods
Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Jiacheng Song,Jiarui Hai,Jiancheng Ye,Zidan Wang

DOI: https://doi.org/10.1101/2023.08.24.23294597

2023-08-25

MedRxiv

Abstract:Objective: To propose a novel approach for enhancing clinical prediction models by combining structured and unstructured data with multimodal data fusion. Methods: We presented a comprehensive framework that integrated multimodal data sources, including textual clinical notes, structured electronic health records (EHRs), and relevant clinical data from National Electronic Injury Surveillance System (NEISS) datasets. We proposed a novel hybrid fusion method, which incorporated state-of-the-art pre-trained language model, to integrate unstructured clinical text with structured EHR data and other multimodal sources, thereby capturing a more comprehensive representation of patient information. Results: The experimental results demonstrated that the hybrid fusion approach significantly improved the performance of clinical prediction models compared to traditional fusion frameworks and unimodal models that rely solely on structured data or text information alone. The proposed hybrid fusion system with RoBERTa language encoder achieved the best prediction of the Top 1 injury with an accuracy of 75.00% and Top 3 injuries with an accuracy of 93.54%. Conclusion: Our study highlights the potential of integrating natural language processing (NLP) techniques with multimodal data fusion for enhancing clinical prediction models' performances. By leveraging the rich information present in clinical text and combining it with structured EHR data, the proposed approach can improve the accuracy and robustness of predictive models. The approach has the potential to advance clinical decision support systems, enable personalized medicine, and facilitate evidence-based health care practices. Future research can further explore the application of this hybrid fusion approach in real-world clinical settings and investigate its impact on improving patient outcomes.

Computer Science,Medicine
PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo,Zhipeng Wang,Ruiquan Ge,Jianxun Yu,Feiwei Qin,Yuan Tian,Yuqing Peng,Yonghong Li,Changmiao Wang

2024-04-17

Abstract:The early detection of a pulmonary embolism (PE) is critical for enhancing patient survival rates. Both image-based and non-image-based features are of utmost importance in medical classification tasks. In a clinical setting, physicians tend to rely on the contextual information provided by Electronic Medical Records (EMR) to interpret medical imaging. However, very few models effectively integrate clinical information with imaging data. To address this shortcoming, we suggest a multimodal fusion methodology, termed PE-MVCNet, which capitalizes on Computed Tomography Pulmonary Angiography imaging and EMR data. This method comprises the Image-only module with an integrated multi-view block, the EMR-only module, and the Cross-modal Attention Fusion (CMAF) module. These modules cooperate to extract comprehensive features that subsequently generate predictions for PE. We conducted experiments using the publicly accessible Stanford University Medical Center dataset, achieving an AUROC of 94.1%, an accuracy rate of 90.2%, and an F1 score of 90.6%. Our proposed model outperforms existing methodologies, corroborating that our multimodal fusion model excels compared to models that use a single data modality. Our source code is available at

Image and Video Processing,Computer Vision and Pattern Recognition
[Evaluation of kidney function].

M. Horio

DOI: https://doi.org/10.2169/NAIKA.96.159

2007-01-10

Abstract:
Combining structured and unstructured data for predictive models: a deep learning approach

Dongdong Zhang,Changchang Yin,Jucheng Zeng,Xiaohui Yuan,Ping Zhang

DOI: https://doi.org/10.1186/s12911-020-01297-6

IF: 3.298

2020-10-29

BMC Medical Informatics and Decision Making

Abstract:Abstract Background The broad adoption of electronic health records (EHRs) provides great opportunities to conduct health care research and solve various clinical problems in medicine. With recent advances and success, methods based on machine learning and deep learning have become increasingly popular in medical informatics. However, while many research studies utilize temporal structured data on predictive modeling, they typically neglect potentially valuable information in unstructured clinical notes. Integrating heterogeneous data types across EHRs through deep learning techniques may help improve the performance of prediction models. Methods In this research, we proposed 2 general-purpose multi-modal neural network architectures to enhance patient representation learning by combining sequential unstructured notes with structured data. The proposed fusion models leverage document embeddings for the representation of long clinical note documents and either convolutional neural network or long short-term memory networks to model the sequential clinical notes and temporal signals, and one-hot encoding for static information representation. The concatenated representation is the final patient representation which is used to make predictions. Results We evaluate the performance of proposed models on 3 risk prediction tasks (i.e. in-hospital mortality, 30-day hospital readmission, and long length of stay prediction) using derived data from the publicly available Medical Information Mart for Intensive Care III dataset. Our results show that by combining unstructured clinical notes with structured data, the proposed models outperform other models that utilize either unstructured notes or structured data only. Conclusions The proposed fusion models learn better patient representation by combining structured and unstructured data. Integrating heterogeneous data types across EHRs helps improve the performance of prediction models and reduce errors.

medical informatics
MedCT-BERT: Multimodal Mortality Prediction using Medical ConvTransformer-BERT Model

Ke Niu,Ke Zhang,Wenjuan Tai,Guoqiang Lu,Yuhang Zhou

DOI: https://doi.org/10.1109/ICTAI59109.2023.00109

2023-11-06

Abstract:In the Intensive Care Unit (ICU), mortality prediction tasks primarily rely on clinical records consisting of patients’ clinical time series data and physicians’ diagnostic opinions. However, due to the irregularity of time series data and clinical records, existing medical multimodal models focus on generating complete and regular data to address this issue, neglecting the impact of generated modality data on multimodal feature fusion. Inaccurate generation of modality data may lead to noise. To overcome this problem, we propose a novel medical multimodal model named MedCT-BERT in this paper. Specifically, the existing models can address the irregularities in time series data but are limited to handling fixed time series data. We optimize their hyperparameters for better performance. By doing so, our model can process and generate complete regular imputed values in parallel for time series data with varying missing values and sequence lengths. To iteratively refine the generated imputed values, we introduce a time series feature correlation information to reduce noise in multimodal data fusion. We conduct experiments on the MIMIC-III dataset with MedCT-BERT and validate the effectiveness of the model in mortality prediction tasks.

Computer Science,Medicine
Enhancing In-Hospital Mortality Prediction Using Multi-Representational Learning with LLM-Generated Expert Summaries

Harshavardhan Battula,Jiacheng Liu,Jaideep Srivastava

2024-11-26

Abstract:In-hospital mortality (IHM) prediction for ICU patients is critical for timely interventions and efficient resource allocation. While structured physiological data provides quantitative insights, clinical notes offer unstructured, context-rich narratives. This study integrates these modalities with Large Language Model (LLM)-generated expert summaries to improve IHM prediction accuracy. Using the MIMIC-III database, we analyzed time-series physiological data and clinical notes from the first 48 hours of ICU admission. Clinical notes were concatenated chronologically for each patient and transformed into expert summaries using Med42-v2 70B. A multi-representational learning framework was developed to integrate these data sources, leveraging LLMs to enhance textual data while mitigating direct reliance on LLM predictions, which can introduce challenges in uncertainty quantification and interpretability. The proposed model achieved an AUPRC of 0.6156 (+36.41%) and an AUROC of 0.8955 (+7.64%) compared to a time-series-only baseline. Expert summaries outperformed clinical notes or time-series data alone, demonstrating the value of LLM-generated knowledge. Performance gains were consistent across demographic groups, with notable improvements in underrepresented populations, underscoring the framework's equitable application potential. By integrating LLM-generated summaries with structured and unstructured data, the framework captures complementary patient information, significantly improving predictive performance. This approach showcases the potential of LLMs to augment critical care prediction models, emphasizing the need for domain-specific validation and advanced integration strategies for broader clinical adoption.

Computation and Language,Artificial Intelligence,Machine Learning

[Pedigree of Waardenburg's syndrome in a family].

Poem: lichenplanus.

Multi-modal learning for inpatient length of stay prediction

Dual-Core Mutual Learning Between Scoring Systems and Clinical Features for ICU Mortality Prediction

Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality Prediction

Fusion of medical imaging and electronic health records with attention and multi-head machanisms

Research on Multimodal Fusion of Temporal Electronic Medical Records

Multi-channel fusion LSTM for medical event prediction using EHRs

DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data.

Monitoring ICU Mortality Risk with A Long Short-Term Memory Recurrent Neural Network

MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Automated Fusion of Multimodal Electronic Health Records for Better Medical Predictions

Self-Supervised Predictive Coding with Multimodal Fusion for Patient Deterioration Prediction in Fine-grained Time Resolution

Prediction of Mortality Events of Patients with Acute Heart Failure in Intensive Care Unit Based on Deep Neural Network

Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

[Evaluation of kidney function].

Combining structured and unstructured data for predictive models: a deep learning approach

MedCT-BERT: Multimodal Mortality Prediction using Medical ConvTransformer-BERT Model

Enhancing In-Hospital Mortality Prediction Using Multi-Representational Learning with LLM-Generated Expert Summaries