Abstract:Depression is a prevalent mental disorder affecting a significant portion of the global population, leading to considerable disability and contributing to the overall burden of disease. Consequently, designing efficient and robust automated methods for depression detection has become imperative. Recently, deep learning methods, especially multimodal fusion methods, have been increasingly used in computer-aided depression detection. Importantly, individuals with depression and those without respond differently to various emotional stimuli, providing valuable information for detecting depression. Building on these observations, we propose an intra- and inter-emotional stimulus transformer-based fusion model to effectively extract depression-related features. The intra-emotional stimulus fusion framework aims to prioritize different modalities, capitalizing on their diversity and complementarity for depression detection. The inter-emotional stimulus model maps each emotional stimulus onto both invariant and specific subspaces using individual invariant and specific encoders. The emotional stimulus-invariant subspace facilitates efficient information sharing and integration across different emotional stimulus categories, while the emotional stimulus specific subspace seeks to enhance diversity and capture the distinct characteristics of individual emotional stimulus categories. Our proposed intra- and inter-emotional stimulus fusion model effectively integrates multimodal data under various emotional stimulus categories, providing a comprehensive representation that allows accurate task predictions in the context of depression detection. We evaluate the proposed model on the Chinese Soochow University students dataset, and the results outperform state-of-the-art models in terms of concordance correlation coefficient (CCC), root mean squared error (RMSE) and accuracy.

A supervised contrastive learning-based model for image emotion classification

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

Semi-Supervised Classification of Emotional Pictures Based on Feature Combination

Learning multi-level representations for affective image recognition

Attention-based multi-level image and text sentiment analysis

A Multi-Stage Visual Perception Approach for Image Emotion Analysis

Progressive Visual Content Understanding Network for Image Emotion Classification

Adaptive Deep Metric Learning for Affective Image Retrieval and Classification

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Object aroused emotion analysis network for image sentiment analysis

Emotion recognition of social media users based on deep learning

Attention-Based BiLSTM Network with Lexical Feature for Emotion Classification

The application of deep learning in computer vision: Facial emotion recognition based on convolutional neural network

Learning Multi-level Deep Representations for Image Emotion Classification

Self-Supervised EEG Representation Learning for Robust Emotion Recognition

Visual Emotion Analysis Via Affective Semantic Concept Discovery

Multimodal Sentiment Analysis of Government Information Comments Based on Contrastive Learning and Cross-Attention Fusion Networks

TSCL-FHFN: two-stage contrastive learning and feature hierarchical fusion network for multimodal sentiment analysis

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

Inferring Emotions from Social Images Leveraging Influence Analysis