Abstract:Mild Cognitive Impairment (MCI) is an early stage of memory loss or other cognitive ability loss in individuals who maintain the ability to independently perform most activities of daily living. It is considered a transitional stage between normal cognitive stage and more severe cognitive declines like dementia or Alzheimer's. Based on the reports from the National Institute of Aging (NIA), people with MCI are at a greater risk of developing dementia, thus it is of great importance to detect MCI at the earliest possible to mitigate the transformation of MCI to Alzheimer's and dementia. Recent studies have harnessed Artificial Intelligence (AI) to develop automated methods to predict and detect MCI. The majority of the existing research is based on unimodal data (e.g., only speech or prosody), but recent studies have shown that multimodality leads to a more accurate prediction of MCI. However, effectively exploiting different modalities is still a big challenge due to the lack of efficient fusion methods. This study proposes a robust fusion architecture utilizing an embedding-level fusion via a co-attention mechanism to leverage multimodal data for MCI prediction. This approach addresses the limitations of early and late fusion methods, which often fail to preserve inter-modal relationships. Our embedding-level fusion aims to capture complementary information across modalities, enhancing predictive accuracy. We used the I-CONECT dataset, where a large number of semi-structured conversations via internet/webcam between participants aged 75+ years old and interviewers were recorded. We introduce a multimodal speech-language-vision Deep Learning-based method to differentiate MCI from Normal Cognition (NC). Our proposed architecture includes co-attention blocks to fuse three different modalities at the embedding level to find the potential interactions between speech (audio), language (transcribed speech), and vision (facial videos) within the cross-Transformer layer. Experimental results demonstrate that our fusion method achieves an average AUC of 85.3% in detecting MCI from NC, significantly outperforming unimodal (60.9%) and bimodal (76.3%) baseline models. This superior performance highlights the effectiveness of our model in capturing and utilizing the complementary information from multiple modalities, offering a more accurate and reliable approach for MCI prediction.

A Transformer Approach for Cognitive Impairment Classification and Prediction

A Transformer-based Multi-features Fusion Model for Prediction of Conversion in Mild Cognitive Impairment

A transformer-based unified multimodal framework for Alzheimer's disease assessment

Prediction Of Mild Cognitive Impairment to Alzheimer’s Disease Conversion Via Machine Learning

Early Diagnosis of Alzheimer's Disease Using Deep Learning

TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data

Generative forecasting of brain activity enhances Alzheimer's classification and interpretation

Predicting Conversion of Mild Cognitive Impairments to Alzheimer's Disease and Exploring Impact of Neuroimaging

Enhancing healthcare recommendation: transfer learning in deep convolutional neural networks for Alzheimer disease detection

Incomplete Multimodal Learning for Complex Brain Disorders Prediction

Diagnosis of Alzheimer's disease via optimized lightweight convolution-attention and structural MRI

Toward Robust Early Detection of Alzheimer's Disease via an Integrated Multimodal Learning Approach

Double-attention Assisted Multi-task Learning for the Alzheimer’s Disease Prediction from Mild Cognitive Impairment*

VGG-TSwinformer: Transformer-based deep learning model for early Alzheimer's disease prediction

An attention-based deep learning approach for the classification of subjective cognitive decline and mild cognitive impairment using resting-state EEG

Transfer Learning and Class Decomposition for Detecting the Cognitive Decline of Alzheimer Disease

Machine Learning of Cerebello-Cerebral Functional Networks for Mild Cognitive Impairment Detection

Enhancing MRI-Based Classification of Alzheimer's Disease with Explainable 3D Hybrid Compact Convolutional Transformers

A multimodal cross-transformer-based model to predict mild cognitive impairment using speech, language and vision

Multi-feature fusion learning for Alzheimer's disease prediction using EEG signals in resting state