Abstract:Mild Cognitive Impairment (MCI) is an early stage of memory loss or other cognitive ability loss in individuals who maintain the ability to independently perform most activities of daily living. It is considered a transitional stage between normal cognitive stage and more severe cognitive declines like dementia or Alzheimer's. Based on the reports from the National Institute of Aging (NIA), people with MCI are at a greater risk of developing dementia, thus it is of great importance to detect MCI at the earliest possible to mitigate the transformation of MCI to Alzheimer's and dementia. Recent studies have harnessed Artificial Intelligence (AI) to develop automated methods to predict and detect MCI. The majority of the existing research is based on unimodal data (e.g., only speech or prosody), but recent studies have shown that multimodality leads to a more accurate prediction of MCI. However, effectively exploiting different modalities is still a big challenge due to the lack of efficient fusion methods. This study proposes a robust fusion architecture utilizing an embedding-level fusion via a co-attention mechanism to leverage multimodal data for MCI prediction. This approach addresses the limitations of early and late fusion methods, which often fail to preserve inter-modal relationships. Our embedding-level fusion aims to capture complementary information across modalities, enhancing predictive accuracy. We used the I-CONECT dataset, where a large number of semi-structured conversations via internet/webcam between participants aged 75+ years old and interviewers were recorded. We introduce a multimodal speech-language-vision Deep Learning-based method to differentiate MCI from Normal Cognition (NC). Our proposed architecture includes co-attention blocks to fuse three different modalities at the embedding level to find the potential interactions between speech (audio), language (transcribed speech), and vision (facial videos) within the cross-Transformer layer. Experimental results demonstrate that our fusion method achieves an average AUC of 85.3% in detecting MCI from NC, significantly outperforming unimodal (60.9%) and bimodal (76.3%) baseline models. This superior performance highlights the effectiveness of our model in capturing and utilizing the complementary information from multiple modalities, offering a more accurate and reliable approach for MCI prediction.

A Transformer-based Multi-features Fusion Model for Prediction of Conversion in Mild Cognitive Impairment

Prediction Of Mild Cognitive Impairment to Alzheimer’s Disease Conversion Via Machine Learning

Predicting Conversion of Mild Cognitive Impairments to Alzheimer's Disease and Exploring Impact of Neuroimaging

Multi-level fusion network for mild cognitive impairment identification using multi-modal neuroimages

TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

Dual attention based fusion network for MCI Conversion Prediction

Predicting conversion of mild cognitive impairment to Alzheimer's disease

VGG-TSwinformer: Transformer-based deep learning model for early Alzheimer's disease prediction

A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision

Brain Connectivity Based Prediction of Alzheimer’s Disease in Patients with Mild Cognitive Impairment Based on Multi-Modal Images

Multi-feature fusion learning for Alzheimer's disease prediction using EEG signals in resting state

A Transformer Approach for Cognitive Impairment Classification and Prediction

Predicting long-term progression of Alzheimer's disease using a multimodal deep learning model incorporating interaction effects

Decision Supporting Model for One-year Conversion Probability from MCI to AD using CNN and SVM.

Heterogeneous data fusion for predicting mild cognitive impairment conversion

A deep feature fusion network with global context and cross-dimensional dependencies for classification of mild cognitive impairment from brain MRI

Transferable Structural, Static and Dynamic Functional MRI Features for Predicting Conversion from Mild Cognitive Impairment to Alzheimer’s Disease in Shanghai Memory Study and ADNI Cohorts

Convolutional Neural Networks-Based MRI Image Analysis for the Alzheimer’s Disease Prediction From Mild Cognitive Impairment

Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline

Multi-view imputation and cross-attention network based on incomplete longitudinal and multimodal data for conversion prediction of mild cognitive impairment

Enhancing Early Alzheimer's Disease Classification Accuracy Through the Fusion of Smri and Rsmeg Data: a Deep Learning Approach