Abstract:At present, although great progress has been made in automatic depression assessment, most of the recent works only concern the audio and video paralinguistic information, rather than the linguistic information from the spoken content. In this work, we argue that beside developing good audio and video features, to build reliable depression detection systems, text-based content features are also of importance to analyse depression-related textual indicators. Furthermore, to improve the performance of automatic depression assessment systems, powerful models, capable of modelling the characteristics of depression embedded in the audio, visual and text descriptors, are also required. This paper proposes new text and video features and hybridizes deep and shallow models for depression estimation and classification from audio, video and text descriptors. The proposed hybrid framework consists of three main parts: 1) A Deep Convolutional Neural Network (DCNN) and Deep Neural Network (DNN) based audio-visual multi-modal depression recognition model for estimating the Patient Health Questionnaire depression scale (PHQ-8); 2) A Paragraph Vector (PV) and Support Vector Machine (SVM) based model for inferring the physical and mental conditions of the individual from the transcripts of the interview; 3) A Random Forest (RF) model for depression classification from the estimated PHQ-8 score and the inferred conditions of the individual. In the PV-SVM model, PV embedding is used to obtain fixed-length feature vectors from transcripts of the answers to the questions associated with psychoanalytic aspects of depression, which are subsequently fed into the SVM classifiers for detecting the presence/absence of the considered psychoanalytic symptoms. To our best knowledge, this approach is the first attempt to apply PV for depression analysis. Besides, we propose a new visual descriptor - Histogram of Displacement Range (HDR) to characterize the displacement and velocity of the facial landmarks in the video segment. Experiments have been carried out on the Audio Visual Emotion Challenge (AVEC2016) depression dataset, they demonstrate that: 1) The proposed hybrid framework effectively improves the accuracies of both depression estimation and depression classification, with an average F1 measure up to 0.746, which is higher than the best result (0.724) of the depression sub-challenge of AVEC2016. 2) HDR obtains better depression recognition performance than Bag-of-Words (BoW) and Motion History Histogram (MHH) features.

Depressformer: Leveraging Video Swin Transformer and fine-grained local features for depression scale estimation

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Automatic Assessment of Depression from Speech Via a Hierarchical Attention Transfer Network and Attention Autoencoders

LSCAformer: Long and short-term cross-attention-aware transformer for depression recognition from video sequences

DepNet: An automated industrial intelligent system using deep learning for video‐based depression analysis

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

Depressioner: Facial dynamic representation for automatic depression level prediction

Interpreting Depression From Question-Wise Long-Term Video Recording of SDS Evaluation

Dual‐task enhanced global–local temporal–spatial network for depression recognition from facial videos

Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals

Integrating Deep and Shallow Models for Multi-Modal Depression Analysis—Hybrid Architectures

Spectral Representation of Behaviour Primitives for Depression Analysis

EMo Transformer: Transformer-Based Depression Detection Via Eye Movements

DepMSTAT: Multimodal Spatio-Temporal Attentional Transformer for Depression Detection

Hybrid Network Feature Extraction for Depression Assessment from Speech

Multi-Scale and Multi-Region Facial Discriminative Representation for Automatic Depression Level Prediction.

FPT-Former: A Flexible Parallel Transformer of Recognizing Depression by Using Audiovisual Expert-Knowledge-Based Multimodal Measures

A Multi-Frame Rate Network with Attention Mechanism for Depression Severity Estimation.

Multimodal Measurement of Depression Using Deep Learning Models

Encoding Visual Behaviors with Attentive Temporal Convolution for Depression Prediction