Abstract:For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal information from Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). To our best knowledge, the previous methods in the literature mostly used hand-crafted features such as cortical thickness, gray matter densities from MRI, or voxel intensities from PET, and then combined these multimodal features by simply concatenating into a long vector or transforming into a higher-dimensional kernel space. In this paper, we propose a novel method for a high-level latent and shared feature representation from neuroimaging modalities via deep learning. Specifically, we use Deep Boltzmann Machine (DBM)(2), a deep network with a restricted Boltzmann machine as a building block, to find a latent hierarchical feature representation from a 3D patch, and then devise a systematic method for a joint feature representation from the paired patches of MRI and PET with a multimodal DBM. To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods. In three binary classification problems of AD vs. healthy Normal Control (NC), MCI vs. NC, and MCI converter vs. MCI non-converter, we obtained the maximal accuracies of 95.35%, 85.67%, and 74.58%, respectively, outperforming the competing methods. By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

Feature-Based Transformer with Incomplete Multimodal Brain Images for Diagnosis of Neurodegenerative Diseases

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease

Multimodal Diagnosis Model of Alzheimer’s Disease Based on Improved Transformer

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

A transformer-based unified multimodal framework for Alzheimer's disease assessment

Multimodal diagnosis model of Alzheimer's disease based on improved Transformer

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

Task-Induced Pyramid and Attention GAN for Multimodal Brain Image Imputation and Classification in Alzheimer's Disease

Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

MMTFN: Multi‐modal multi‐scale transformer fusion network for Alzheimer's disease diagnosis

Collaborative Image Synthesis and Disease Diagnosis for Classification of Neurodegenerative Disorders with Incomplete Multi-modal Neuroimages

Alzheimer's Disease Prediction via Brain Structural-Functional Deep Fusing Network

Tensor-Based Multi-Modality Feature Selection and Regression for Alzheimer's Disease Diagnosis

3D Transformer based on deformable patch location for differential diagnosis between Alzheimer's disease and Frontotemporal dementia

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and

Multimodal Cross Enhanced Fusion Network for Diagnosis of Alzheimer's Disease and Subjective Memory Complaints.

Multimodal 3D Convolutional Neural Networks for Classification of Brain Disease Using Structural MR and FDG-PET Images.

Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

Multiple Inputs and Mixed Data for Alzheimer's Disease Classification Based on 3D Vision Transformer

A Hybrid Multi-Scale Attention Convolution and Aging Transformer Network for Alzheimer's Disease Diagnosis.