Abstract:Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.

Text-based Decision Fusion Model for Detecting Depression

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

A Multimodal Approach for Detection and Assessment of Depression Using Text, Audio and Video

Enhancing depression detection: A multimodal approach with text extension and content fusion

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Fusing Multi-Level Features from Audio and Contextual Sentence Embedding from Text for Interview-Based Depression Detection

Multimodal Depression Detection: Fusion of Electroencephalography and Paralinguistic Behaviors Using a Novel Strategy for Classifier Ensemble.

Towards Automatic Depression Detection: A BiLSTM/1D CNN-Based Model

A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Fusing features of speech for depression classification based on higher-order spectral analysis

An Intelligent Depression Detection Model Based On Multimodal Fusion Technology

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Text-based Depression Detection: What Triggers an Alert

Investigating Acoustic-Textual Emotional Inconsistency Information for Automatic Depression Detection

Adolescent Depression Detection Model Based on Multimodal Data of Interview Audio and Text.

Attention-Based Acoustic Feature Fusion Network for Depression Detection

A Multi-modal Feature Layer Fusion Model for Assessment of Depression Based on Attention Mechanisms

A Novel Decision Tree for Depression Recognition in Speech

Feature-level fusion approaches based on multimodal EEG data for depression recognition