DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET

Yitong Li,Morteza Ghahremani,Youssef Wally,Christian Wachinger

2024-10-31

Abstract:Diagnosing dementia, particularly for Alzheimer's Disease (AD) and frontotemporal dementia (FTD), is complex due to overlapping symptoms. While magnetic resonance imaging (MRI) and positron emission tomography (PET) data are critical for the diagnosis, integrating these modalities in deep learning faces challenges, often resulting in suboptimal performance compared to using single modalities. Moreover, the potential of multi-modal approaches in differential diagnosis, which holds significant clinical importance, remains largely unexplored. We propose a novel framework, DiaMond, to address these issues with vision Transformers to effectively integrate MRI and PET. DiaMond is equipped with self-attention and a novel bi-attention mechanism that synergistically combine MRI and PET, alongside a multi-modal normalization to reduce redundant dependency, thereby boosting the performance. DiaMond significantly outperforms existing multi-modal methods across various datasets, achieving a balanced accuracy of 92.4% in AD diagnosis, 65.2% for AD-MCI-CN classification, and 76.5% in differential diagnosis of AD and FTD. We also validated the robustness of DiaMond in a comprehensive ablation study. The code is available at <a class="link-external link-https" href="https://github.com/ai-med/DiaMond" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the diagnostic challenges of Alzheimer's disease (AD) and other types of dementia, such as frontotemporal dementia (FTD). Specifically, the paper focuses on the following aspects: 1. **Overlap of complex symptoms**: The symptoms of dementias like AD and FTD overlap, making it difficult to accurately distinguish between different types of dementia based solely on clinical presentation. 2. **Challenges of multimodal data integration**: Although magnetic resonance imaging (MRI) and positron emission tomography (PET) are valuable in diagnosing dementia, effectively integrating these multimodal data into deep learning models faces many challenges, often resulting in performance that is inferior to unimodal methods. 3. **Importance of differential diagnosis**: Most current research focuses on a single type of dementia (mainly AD), while differential diagnosis (i.e., distinguishing between different types of dementia) is clinically significant but less studied. To address these issues, the paper proposes a new framework—DiaMond, which effectively integrates MRI and PET data using Vision Transformers. DiaMond significantly improves the accuracy of dementia diagnosis through self-attention mechanisms and a novel dual-attention mechanism, combined with multimodal normalization techniques, and excels particularly in the differential diagnosis of AD and FTD.

DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers Using MRI and PET

Dominating Alzheimer's disease diagnosis with deep learning on sMRI and DTI-MD

Dual-3DM3AD: Mixed Transformer Based Semantic Segmentation and Triplet Pre-Processing for Early Multi-Class Alzheimer's Diagnosis

Diagnosis of Alzheimer's disease via optimized lightweight convolution-attention and structural MRI

Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline

HAMMF: Hierarchical Attention-Based Multi-Task and Multi-Modal Fusion Model for Computer-Aided Diagnosis of Alzheimer’s Disease

Multi-modal cross-attention network for Alzheimer's disease diagnosis with multi-modality data

MMTFN: Multi‐modal multi‐scale transformer fusion network for Alzheimer's disease diagnosis

Toward Robust Early Detection of Alzheimer's Disease via an Integrated Multimodal Learning Approach

Multiple Inputs and Mixed Data for Alzheimer's Disease Classification Based on 3D Vision Transformer

Multimodal and Multiscale Deep Neural Networks for the Early Diagnosis of Alzheimer's Disease using structural MR and FDG-PET images

HAMMF: Hierarchical attention-based multi-task and multi-modal fusion model for computer-aided diagnosis of Alzheimer's disease

Recognition of Alzheimer's disease and Mild Cognitive Impairment with multimodal image-derived biomarkers and Multiple Kernel Learning

Multimodal Cross Enhanced Fusion Network for Diagnosis of Alzheimer's Disease and Subjective Memory Complaints.

Tensor-Based Multi-Modality Feature Selection and Regression for Alzheimer's Disease Diagnosis

Patch-based Interpretable Deep Learning Framework for Alzheimer’s Disease Diagnosis Using Multimodal Data

Multimodal cross enhanced fusion network for diagnosis of Alzheimer?s disease and

Automated detection of Alzheimer's disease: a multi-modal approach with 3D MRI and amyloid PET

An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer's Disease Diagnosis

A transformer-based unified multimodal framework for Alzheimer's disease assessment

ADAPT: Alzheimer Diagnosis through Adaptive Profiling Transformers