Abstract:Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.

LSCAformer: Long and short-term cross-attention-aware transformer for depression recognition from video sequences

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Automatic Assessment of Depression from Speech Via a Hierarchical Attention Transfer Network and Attention Autoencoders

Depressformer: Leveraging Video Swin Transformer and fine-grained local features for depression scale estimation

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Dual‐task enhanced global–local temporal–spatial network for depression recognition from facial videos

Neural Architecture Searching for Facial Attributes-based Depression Recognition

Prediction of Depression Severity Based on Transformer Encoder and CNN Model

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Spectral Representation of Behaviour Primitives for Depression Analysis

Two Birds With One Stone: Knowledge-Embedded Temporal Convolutional Transformer for Depression Detection and Emotion Recognition

Hierarchical Attention Transfer Networks for Depression Assessment from Speech

Depressioner: Facial dynamic representation for automatic depression level prediction

An Improved Global-Local Fusion Network for Depression Detection Telemedicine Framework

DepNet: An automated industrial intelligent system using deep learning for video‐based depression analysis

TCEDN: A Lightweight Time-Context Enhanced Depression Detection Network

Multi-Scale and Multi-Region Facial Discriminative Representation for Automatic Depression Level Prediction.

FacialPulse: An Efficient RNN-based Depression Detection via Temporal Facial Landmarks

SAD-TIME: a Spatiotemporal-fused network for depression detection with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor

Automatic Depression Detection via Learning and Fusing Features from Visual Cues