Abstract:Introduction: Brain degeneration is commonly caused by some chronic diseases, such as Alzheimer's disease (AD) and diabetes mellitus (DM). The risk prediction of brain degeneration aims to forecast the situation of disease progression of patients in the near future based on their historical health records. It is beneficial for patients to make an accurate clinical diagnosis and early prevention of disease. Current risk predictions of brain degeneration mainly rely on single-modality medical data, such as Electronic Health Records (EHR) or magnetic resonance imaging (MRI). However, only leveraging EHR or MRI data for the pertinent and accurate prediction is insufficient because of single-modality information (e.g., pixel or volume information of image data or clinical context information of non-image data). Methods: Several deep learning-based methods have used multimodal data to predict the risks of specified diseases. However, most of them simply integrate different modalities in an early, intermediate, or late fusion structure and do not care about the intra-modal and intermodal dependencies. A lack of these dependencies would lead to sub-optimal prediction performance. Thus, we propose an encoder-decoder framework for better risk prediction of brain degeneration by using MRI and EHR. An encoder module is one of the key components and mainly focuses on feature extraction of input data. Specifically, we introduce an encoder module, which integrates intra-modal and inter-modal dependencies with the spatial-temporal attention and cross-attention mechanism. The corresponding decoder module is another key component and mainly parses the features from the encoder. In the decoder module, a disease-oriented module is used to extract the most relevant disease representation features. We take advantage of a multi-head attention module followed by a fully connected layer to produce the predicted results. Results: As different types of AD and DM influence the nature and severity of brain degeneration, we evaluate the proposed method for three-class prediction of AD and three-class prediction of DM. Our results show that the proposed method with integrated MRI and EHR data achieves an accuracy of 0.859 and 0.899 for the risk prediction of AD and DM, respectively. Discussion: The prediction performance is significantly better than the benchmarks, including MRI-only, EHR-only, and state-of-the-art multimodal fusion methods.

Recurrent Convolutional Neural Network Based Multimodal Disease Risk Prediction.

Deep Feature Learning for Disease Risk Assessment Based on Convolutional Neural Network with Intra-Layer Recurrent Connection by Using Hospital Big Data.

Prediction of Chronic Kidney Disease Risk Using Multimodal Data

A Method for the Early Prediction of Chronic Diseases Based on Short Sequential Medical Data.

Disease Prediction by Machine Learning Over Big Data From Healthcare Communities

Multimodal risk prediction with physiological signals, medical images and clinical notes

Exploiting Convolutional Neural Network for Risk Prediction with Medical Feature Embedding

Predicting Clinical Visits Using Recurrent Neural Networks and Demographic Information.

Predicting multiple sclerosis disease severity with multimodal deep neural networks

NMCMDA: Neural Multicategory MiRNA-disease Association Prediction.

Multi-disease prediction using LSTM recurrent neural networks

Integrated Convolutional and Recurrent Neural Networks for Health Risk Prediction using Patient Journey Data with Many Missing Values

A Dynamic Deep Neural Network For Multimodal Clinical Data Analysis

A Multimodal Deep Neural Network for Human Breast Cancer Prognosis Prediction by Integrating Multi-Dimensional Data

Improved deep convolutional neural network-based COOT optimization for multimodal disease risk prediction

Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data

Comorbidity and multimorbidity prediction of major chronic diseases using machine learning and network analytics

Diagnosis Prediction Via Recurrent Neural Networks

Multimodal Learning for Cardiovascular Risk Prediction using EHR Data

Deep Featured Adaptive Dense Net Convolutional Neural Network Based Cardiac Risk Prediction in Big Data Healthcare Environment