Abstract:Alzheimer's dementia (AD) entails negative psychological, social, and economic consequences not only for the patients but also for their families, relatives, and society in general. Despite the significance of this phenomenon and the importance for an early diagnosis, there are still limitations. Specifically, the main limitation is pertinent to the way the modalities of speech and transcripts are combined in a single neural network. Existing research works add/concatenate the image and text representations, employ majority voting approaches or average the predictions after training many textual and speech models separately. To address these limitations, in this article we present some new methods to detect AD patients and predict the Mini-Mental State Examination (MMSE) scores in an end-to-end trainable manner consisting of a combination of BERT, Vision Transformer, Co-Attention, Multimodal Shifting Gate, and a variant of the self-attention mechanism. Specifically, we convert audio to Log-Mel spectrograms, their delta, and delta-delta (acceleration values). First, we pass each transcript and image through a BERT model and Vision Transformer, respectively, adding a co-attention layer at the top, which generates image and word attention simultaneously. Secondly, we propose an architecture, which integrates multimodal information to a BERT model via a Multimodal Shifting Gate. Finally, we introduce an approach to capture both the inter- and intra-modal interactions by concatenating the textual and visual representations and utilizing a self-attention mechanism, which includes a gate model. Experiments conducted on the ADReSS Challenge dataset indicate that our introduced models demonstrate valuable advantages over existing research initiatives achieving competitive results in both the AD classification and MMSE regression task. Specifically, our best performing model attains an accuracy of 90.00% and a Root Mean Squared Error (RMSE) of 3.61 in the AD classification task and MMSE regression task, respectively, achieving a new state-of-the-art performance in the MMSE regression task.

Detecting Alzheimer's Disease from Continuous Speech Using Language Models.

Identification of Alzheimer's Disease Patients Based on Oral Speech Features

Leveraging Large Language Models for Identifying Interpretable Linguistic Markers and Enhancing Alzheimer's Disease Diagnostics

Explainable Alzheimer's Disease Detection Using Linguistic Features from Automatic Speech Recognition

Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models

Exploring Multimodal Approaches for Alzheimer's Disease Detection Using Patient Speech Transcript and Audio Data

Leveraging Prompt Learning and Pause Encoding for Alzheimer's Disease Detection

A Transfer Learning Method for Detecting Alzheimer's Disease Based on Speech and Natural Language Processing

Exploring linguistic feature and model combination for speech recognition based automatic AD detection

Profiling Patient Transcript Using Large Language Model Reasoning Augmentation for Alzheimer's Disease Detection

Detecting Alzheimer's Disease Using Natural Language Processing of Referential Communication Task Transcripts

An approach for assisting diagnosis of Alzheimer's disease based on natural language processing

Automatic Identification of Alzheimer's Disease using Lexical Features extracted from Language Samples

Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection

Multimodal Deep Learning Models for Detecting Dementia From Speech and Transcripts

Alzheimer's Disease Detection from Spontaneous Speech through Combining Linguistic Complexity and (Dis)Fluency Features with Pretrained Language Models

Analysis of Speech Features in Alzheimer's Disease with Machine Learning: A Case-Control Study

Automatic speech analysis for detecting cognitive decline of older adults

Towards Within-Class Variation in Alzheimer's Disease Detection from Spontaneous Speech

Noninvasive automatic detection of Alzheimer's disease from spontaneous speech: a review

Leveraging Pretrained Representations with Task-Related Keywords for Alzheimer’s Disease Detection