Abstract:Depression is a prevalent mental disorder affecting a significant portion of the global population, leading to considerable disability and contributing to the overall burden of disease. Consequently, designing efficient and robust automated methods for depression detection has become imperative. Recently, deep learning methods, especially multimodal fusion methods, have been increasingly used in computer-aided depression detection. Importantly, individuals with depression and those without respond differently to various emotional stimuli, providing valuable information for detecting depression. Building on these observations, we propose an intra- and inter-emotional stimulus transformer-based fusion model to effectively extract depression-related features. The intra-emotional stimulus fusion framework aims to prioritize different modalities, capitalizing on their diversity and complementarity for depression detection. The inter-emotional stimulus model maps each emotional stimulus onto both invariant and specific subspaces using individual invariant and specific encoders. The emotional stimulus-invariant subspace facilitates efficient information sharing and integration across different emotional stimulus categories, while the emotional stimulus specific subspace seeks to enhance diversity and capture the distinct characteristics of individual emotional stimulus categories. Our proposed intra- and inter-emotional stimulus fusion model effectively integrates multimodal data under various emotional stimulus categories, providing a comprehensive representation that allows accurate task predictions in the context of depression detection. We evaluate the proposed model on the Chinese Soochow University students dataset, and the results outperform state-of-the-art models in terms of concordance correlation coefficient (CCC), root mean squared error (RMSE) and accuracy.

AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition

AVEC 2018 Workshop and Challenge: Bipolar Disorder and Cross-Cultural Affect Recognition

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Automatic Assessment of Depression from Speech Via a Hierarchical Attention Transfer Network and Attention Autoencoders

Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions

Depression Scale Recognition from Audio, Visual and Text Analysis

Multi-Modal Audio, Video and Physiological Sensor Learning for Continuous Emotion Prediction

Evaluating Acoustic and Linguistic Features of Detecting Depression Sub-Challenge Dataset

Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities

Multimodal Depression Recognition with Dynamic Visual and Audio Cues

Multimodal and Multiresolution Depression Detection from Speech and Facial Landmark Features

Towards Robust Deep Neural Networks for Affect and Depression Recognition from Speech

Exploring Multimodal Visual Features For Continuous Affect Recognition

Internet of Emotional People: Towards Continual Affective Computing Cross Cultures Via Audiovisual Signals.

The INTERSPEECH 2018 Computational Paralinguistics Challenge: Atypical & Self-Assessed Affect, Crying & Heart Beats

Depression Severity Estimation from Multiple Modalities

Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challenge

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Multi-modality Depression Detection Via Multi-scale Temporal Dilated CNNs

Detect Depression from Communication: How Computer Vision, Signal Processing, and Sentiment Analysis Join Forces