Abstract:Depression stands as one of the most widespread psychological disorders and has garnered increasing attention. Currently, how to effectively achieve automatic multimodal depression detection for assisting doctors in early diagnosis of depression, has become an important and challenging issue. To address this issue, this work proposes Transformer-based feature enhancement networks for multimodal depression detection. The proposed method effectively integrates three modalities including video, audio and remote photoplethysmographic (rPPG) signals for multimodal depression detection, in which the rPPG modality is introduced as an additional modality for enhancing the effectiveness of multimodal depression detection. The proposed method consists of three key steps: multimodal feature extraction for video, audio and rPPG modalities, Transformer-based multimodal feature enhancement (TMFE), and graph fusion networks (GFN) based multimodal fusion and depression prediction. More specially, in the stage of multimodal feature extraction, for video and audio modalities we employ deep convolutional neural networks (CNN) to extract the corresponding high-level video and audio features, respectively. For rPPG modality, we adopt a short-time end-to-end rPPG estimation framework to extract the rPPG signal values.The TMFE module stacks multiple Transformers such as the inter-modal, intra-modal, and tri-modal Transformers to jointly capture the dynamics and relationships within and between modalities for each time-step of input sequences. The GFN module is designed to effectively fuse the obtained feature representations from different modalities while maintaining the interactions between them simultaneously. Finally, the obtained shared feature representations of all modalities are fed into a multilayer perceptrons (MLP) network to implement final depression detection tasks. Extensive experiments are conducted on two public datasets such as AVEC2013 and AVEC2014, and experimental results demonstrate the validity of the proposed method on depression detection tasks.

TensorFormer: A Tensor-Based Multimodal Transformer for Multimodal Sentiment Analysis and Depression Detection

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

CubeMLP: an MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Mmformer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation

TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis

Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals

Multimodal Sentiment Analysis Based on Transformer and Low-rank Fusion

TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Two Birds With One Stone: Knowledge-Embedded Temporal Convolutional Transformer for Depression Detection and Emotion Recognition

A Two-Stage Stacked Transformer Framework for Multimodal Sentiment Analysis

SS-Trans (Single-Stream Transformer for Multimodal Sentiment Analysis and Emotion Recognition): The Emotion Whisperer—A Single-Stream Transformer for Multimodal Sentiment Analysis

MEDT: Using Multimodal Encoding-Decoding Network as in Transformer for Multimodal Sentiment Analysis

Tensor Fusion Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

A transformer-encoder-based multimodal multi-attention fusion network for sentiment analysis

Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videos

Transformer-Based Interactive Multi-Modal Attention Network for Video Sentiment Detection

Tri-CLT: Learning Tri-Modal Representations with Contrastive Learning and Transformer for Multimodal Sentiment Recognition