Abstract:Depression is a prevalent mental disorder affecting a significant portion of the global population, leading to considerable disability and contributing to the overall burden of disease. Consequently, designing efficient and robust automated methods for depression detection has become imperative. Recently, deep learning methods, especially multimodal fusion methods, have been increasingly used in computer-aided depression detection. Importantly, individuals with depression and those without respond differently to various emotional stimuli, providing valuable information for detecting depression. Building on these observations, we propose an intra- and inter-emotional stimulus transformer-based fusion model to effectively extract depression-related features. The intra-emotional stimulus fusion framework aims to prioritize different modalities, capitalizing on their diversity and complementarity for depression detection. The inter-emotional stimulus model maps each emotional stimulus onto both invariant and specific subspaces using individual invariant and specific encoders. The emotional stimulus-invariant subspace facilitates efficient information sharing and integration across different emotional stimulus categories, while the emotional stimulus specific subspace seeks to enhance diversity and capture the distinct characteristics of individual emotional stimulus categories. Our proposed intra- and inter-emotional stimulus fusion model effectively integrates multimodal data under various emotional stimulus categories, providing a comprehensive representation that allows accurate task predictions in the context of depression detection. We evaluate the proposed model on the Chinese Soochow University students dataset, and the results outperform state-of-the-art models in terms of concordance correlation coefficient (CCC), root mean squared error (RMSE) and accuracy.

Feature Dimensionality Reduction for Video Affect Classification: A Comparative Study

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Affective Video Classification Based on Spatio-temporal Feature Fusion

Video Classification and Recommendation Based on Affective Analysis of Viewers

Multimodal Deep Denoise Framework for Affective Video Content Analysis.

Automatic facial expression recognition for affective computing based on bag of distances.

Affective Video Content Analysis: Decade Review and New Perspectives

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition.

Multi-View Exclusive Unsupervised Dimension Reduction for Video-Based Facial Expression Recognition

Music video affective understanding using feature importance analysis

Video affective content analysis: a survey of state of the art methods

Multimodal Affective Computing Based on Weighted Linear Fusion

Identifying affective levels on music video via completing the missing modality

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features

A Dimensional Emotion Model Driven Multi-stage Classification of Emotional Speech

Deep learning based affective computing

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

Fine-Grained Facial Expression Analysis Using Dimensional Emotion Model

Affect Estimation in 3D Space Using Multi-Task Active Learning for Regression

Human Color Perception: The Impact of Color Perception on Fine-Grained Emotion Prediction in Movie and Television Videos