Abstract:Introduction: Brain degeneration is commonly caused by some chronic diseases, such as Alzheimer's disease (AD) and diabetes mellitus (DM). The risk prediction of brain degeneration aims to forecast the situation of disease progression of patients in the near future based on their historical health records. It is beneficial for patients to make an accurate clinical diagnosis and early prevention of disease. Current risk predictions of brain degeneration mainly rely on single-modality medical data, such as Electronic Health Records (EHR) or magnetic resonance imaging (MRI). However, only leveraging EHR or MRI data for the pertinent and accurate prediction is insufficient because of single-modality information (e.g., pixel or volume information of image data or clinical context information of non-image data). Methods: Several deep learning-based methods have used multimodal data to predict the risks of specified diseases. However, most of them simply integrate different modalities in an early, intermediate, or late fusion structure and do not care about the intra-modal and intermodal dependencies. A lack of these dependencies would lead to sub-optimal prediction performance. Thus, we propose an encoder-decoder framework for better risk prediction of brain degeneration by using MRI and EHR. An encoder module is one of the key components and mainly focuses on feature extraction of input data. Specifically, we introduce an encoder module, which integrates intra-modal and inter-modal dependencies with the spatial-temporal attention and cross-attention mechanism. The corresponding decoder module is another key component and mainly parses the features from the encoder. In the decoder module, a disease-oriented module is used to extract the most relevant disease representation features. We take advantage of a multi-head attention module followed by a fully connected layer to produce the predicted results. Results: As different types of AD and DM influence the nature and severity of brain degeneration, we evaluate the proposed method for three-class prediction of AD and three-class prediction of DM. Our results show that the proposed method with integrated MRI and EHR data achieves an accuracy of 0.859 and 0.899 for the risk prediction of AD and DM, respectively. Discussion: The prediction performance is significantly better than the benchmarks, including MRI-only, EHR-only, and state-of-the-art multimodal fusion methods.

Beyond Doctors: Future Health Prediction from Multimedia and Multimodal Observations.

A Method for the Early Prediction of Chronic Diseases Based on Short Sequential Medical Data.

Modeling Disease Progression via Multisource Multitask Learners: A Case Study With Alzheimer's Disease.

Multimodal Genotype and Phenotype Data Integration to Improve Partial Data-Based Longitudinal Prediction

Multiview Deep Learning-based Efficient Medical Data Management for Survival Time Forecasting

Visual Analysis of Biomarkers Selected Via Multi-Task Learning for Modeling Alzheimer's Disease Progression

Multimodal risk prediction with physiological signals, medical images and clinical notes

Modeling Disease Progression Flexibly with Nonlinear Disease Structure Via Multi-task Learning

ChroNet: A Multi-Task Learning Based Approach for Prediction of Multiple Chronic Diseases

Healthcare Predictive Analytics for Disease Progression: a Longitudinal Data Fusion Approach

Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modeling

A Small Target Segmentation-based Assistive Medical Diagnosis Method via Multimedia Data Analysis

Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future Directions

Rethinking modeling Alzheimer's disease progression from a multi-task learning perspective with deep recurrent neural network

Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data

Modeling disease progression via multi-task learning

Multi-modal Sequence Learning for Alzheimer's Disease Progression Prediction with Incomplete Variable-Length Longitudinal Data.

A Hierarchical Attention-Based Multimodal Fusion Framework for Predicting the Progression of Alzheimer’s Disease

Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review

Deep EHR: Chronic Disease Prediction Using Medical Notes

Multi-modal sequence learning for Alzheimer’s disease progression prediction with incomplete variable-length longitudinal data