Abstract:In recent years, deep learning models have been applied to neuroimaging data for early diagnosis of Alzheimer's disease (AD). Structural magnetic resonance imaging (sMRI) and positron emission tomography (PET) images provide structural and functional information about the brain, respectively. Combining these features leads to improved performance than using a single modality alone in building predictive models for AD diagnosis. However, current multi-modal approaches in deep learning, based on sMRI and PET, are mostly limited to convolutional neural networks, which do not facilitate integration of both image and phenotypic information of subjects. We propose to use graph neural networks (GNN) that are designed to deal with problems in non-Euclidean domains. In this study, we demonstrate how brain networks can be created from sMRI or PET images and be used in a population graph framework that can combine phenotypic information with imaging features of these brain networks. Then, we present a multi-modal GNN framework where each modality has its own branch of GNN and a technique is proposed to combine the multi-modal data at both the level of node vectors and adjacency matrices. Finally, we perform late fusion to combine the preliminary decisions made in each branch and produce a final prediction. As multi-modality data becomes available, multi-source and multi-modal is the trend of AD diagnosis. We conducted explorative experiments based on multi-modal imaging data combined with non-imaging phenotypic information for AD diagnosis and analyzed the impact of phenotypic information on diagnostic performance. Results from experiments demonstrated that our proposed multi-modal approach improves performance for AD diagnosis, and this study also provides technical reference and support the need for multivariate multi-modal diagnosis methods.

Characterizing and Understanding End-to-End Multi-Modal Neural Networks on GPUs.

MMBench: Benchmarking End-to-End Multi-modal DNNs and Understanding Their Hardware-Software Implications

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Characterizing and Understanding HGNN Training on GPUs

Characterizing and Understanding GCNs on GPU

Towards Universal Performance Modeling for Machine Learning Training on Multi-GPU Platforms

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

Characterizing and Efficiently Accelerating Multimodal Generation Model Inference

Fluorescein labeling of tissue plasminogen activator for subretinal thrombolysis.

A Survey of Multi-Tenant Deep Learning Inference on GPU

MMNet: Multi-modal multi-stage network for RGB-T image semantic segmentation

MSTGC: Multi-Channel Spatio-Temporal Graph Convolution Network for Multi-Modal Brain Networks Fusion

Multi-path Neural Networks for On-device Multi-domain Visual Classification

Multi-modal Graph Neural Network for Early Diagnosis of Alzheimer's Disease from sMRI and PET Scans

Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond

Characterizing the I/O Pipeline in the Deployment of CNNs on Commercial Accelerators

Dynamic Multi-path Neural Network

Implementation and evaluation of deep neural networks (DNN) on mainstream heterogeneous systems

Multiplex Graph Networks for Multimodal Brain Network Analysis

Distributed Foundation Models for Multi-Modal Learning in 6G Wireless Networks