Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

Linfeng Liu,Siyu Liu,Lu Zhang,Xuan Vinh To,Fatima Nasrallah,Shekhar S. Chandra

2023-07-16

Abstract:Accurate medical classification requires a large number of multi-modal data, and in many cases, different feature types. Previous studies have shown promising results when using multi-modal data, outperforming single-modality models when classifying diseases such as Alzheimer's Disease (AD). However, those models are usually not flexible enough to handle missing modalities. Currently, the most common workaround is discarding samples with missing modalities which leads to considerable data under-utilization. Adding to the fact that labeled medical images are already scarce, the performance of data-driven methods like deep learning can be severely hampered. Therefore, a multi-modal method that can handle missing data in various clinical settings is highly desirable. In this paper, we present Multi-Modal Mixing Transformer (3MAT), a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. In this work, we test 3MT for AD and Cognitively normal (CN) classification and mild cognitive impairment (MCI) conversion prediction to progressive MCI (pMCI) or stable MCI (sMCI) using clinical and neuroimaging data. The model uses a novel Cascaded Modality Transformer architecture with cross-attention to incorporate multi-modal information for more informed predictions. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios. The result is a versatile network that enables the mixing of arbitrary numbers of modalities with different feature types and also ensures full data utilization missing data scenarios. The model is trained and evaluated on the ADNI dataset with the SOTRA performance and further evaluated with the AIBL dataset with missing data.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is dealing with the situation of incomplete multimodal data in Alzheimer's disease (AD) classification and mild cognitive impairment (MCI) conversion prediction. Specifically, the existing multimodal models are not flexible enough when dealing with missing modalities. The common practice is to discard samples with missing modalities, which leads to a great waste of data. In addition, since labeled medical images are scarce in themselves, this practice will seriously affect the performance of data - driven methods (such as deep learning). To overcome these problems, the author proposes a new model named Multi - Modal Mixing Transformer (3MT). This model can not only utilize multimodal data, but also is specially designed with the ability to handle missing - data scenarios. By introducing a novel cascaded - modality - transformer architecture and cross - attention mechanism, 3MT can integrate multimodal information to make more accurate predictions. Moreover, a new Modality Dropout (MDrop) mechanism is proposed in the paper, ensuring that the model has unprecedented modality independence and robustness when dealing with missing - data scenarios. This enables 3MT to efficiently handle modal data of any number and different feature types in various clinical settings while ensuring full utilization of data. In summary, the main goal of this paper is to develop a disease - classification model that can still work effectively in the case of incomplete multimodal data, thereby improving the diagnostic accuracy of Alzheimer's disease and other related diseases.

Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease

Multi-modal cross-attention network for Alzheimer's disease diagnosis with multi-modality data

MMTFN: Multi‐modal multi‐scale transformer fusion network for Alzheimer's disease diagnosis

MDMA: Multimodal Data and Multi-attention Based Deep Learning Model for Alzheimer’s Disease Diagnosis

HAMMF: Hierarchical Attention-Based Multi-Task and Multi-Modal Fusion Model for Computer-Aided Diagnosis of Alzheimer’s Disease

ADMultiImg: a Novel Missing Modality Transfer Learning Based CAD System for Diagnosis of MCI Due to AD Using Incomplete Multi-Modality Imaging Data.

Multi-Modality Cascaded Convolutional Neural Networks for Alzheimer’s Disease Diagnosis

Multi-scale multimodal deep learning framework for Alzheimer's disease diagnosis

Latent Representation Learning for Alzheimer's Disease Diagnosis with Incomplete Multi-Modality Neuroimaging and Genetic Data.

HAMMF: Hierarchical attention-based multi-task and multi-modal fusion model for computer-aided diagnosis of Alzheimer's disease

Multimodal Diagnosis Model of Alzheimer’s Disease Based on Improved Transformer

Feature-Based Transformer with Incomplete Multimodal Brain Images for Diagnosis of Neurodegenerative Diseases

Multi-modal latent space inducing ensemble SVM classifier for early dementia diagnosis with neuroimaging data

A transformer-based unified multimodal framework for Alzheimer's disease assessment

A Multi-classification Accessment Framework for Reproducible Evaluation of Multimodal Learning in Alzheimer's Disease

Dual-3DM3AD: Mixed Transformer Based Semantic Segmentation and Triplet Pre-Processing for Early Multi-Class Alzheimer's Diagnosis

Multimodal deep learning models for early detection of Alzheimer’s disease stage

Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis

Subclass-based multi-task learning for Alzheimer's disease diagnosis

Multiple Inputs and Mixed Data for Alzheimer's Disease Classification Based on 3D Vision Transformer