Abstract:Multi-modal emotion analysis, as an important direction in affective computing, has attracted increasing attention in recent years. Most existing multi-modal emotion recognition studies are targeted at a classification task that aims to assign a specific emotion category to a combination of several heterogeneous input data, including multimedia signals and physiological signals. Compared to single-class emotion recognition, a growing number of recent psychological evidence suggests that different discrete emotions may co-exist at the same time, which promotes the development of mixed-emotion recognition to identify a mixture of basic emotions. Although most current studies treat it as a multi-label classification task, in this work, we focus on a challenging situation where both positive and negative emotions are presented simultaneously, and propose a multi-modal mixed emotion recognition framework, namely EmotionDict. The key characteristics of our EmotionDict include the following. (1) Inspired by the psychological evidence that such a mixed state can be represented by combinations of basic emotions, we address mixed emotion recognition as a label distribution learning task. An emotion dictionary has been designed to disentangle the mixed emotion representations into a weighted sum of a set of basic emotion elements in a shared latent space and their corresponding weights. (2) While many existing emotion distribution studies are built on a single type of multimedia signal (such as text, image, audio, and video), we incorporate physiological and overt behavioral multi-modal signals, including electroencephalogram (EEG), peripheral physiological signals, and facial videos, which directly display the subjective emotions. These modalities have diverse characteristics given that they are related to the central or peripheral nervous system, and the motor cortex. (3) We further design auxiliary tasks to learn modality attentions for modality integration. Experiments on two datasets show that our method outperforms existing state-of-the-art approaches on mixed-emotion recognition.

Multiple Emotional Tagging of Multimedia Data by Exploiting Dependencies among Emotions.

Capturing Emotion Distribution for Multimedia Emotion Tagging

Multiple Emotion Tagging for Multimedia Data by Exploiting High-Order Dependencies Among Emotions

Capturing Dependencies among Labels and Features for Multiple Emotion Tagging of Multimedia Data.

Emotional Tagging Of Videos By Exploring Multiple Emotions' Coexistence

Exploiting Multi-Expression Dependences for Implicit Multi-Emotion Video Tagging.

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Implicit Video Multi-Emotion Tagging by Exploiting Multi-Expression Relations.

Exploiting Multi-Emotion Relations at Feature and Label Levels for Emotion Tagging

Content-Based Video Emotion Tagging Augmented by Users’ Multiple Physiological Responses

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

Implicit Hybrid Video Emotion Tagging By Integrating Video Content And Users' Multiple Physiological Responses

Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging

Exploiting EEG signals and audiovisual feature fusion for video emotion recognition

Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing

Emotion Dictionary Learning with Modality Attentions for Mixed Emotion Exploration

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

Implicit Video Emotion Tagging from Audiences’ Facial Expression

Semi-supervised Bayesian Deep Multi-modal Emotion Recognition.

Emotional Semantic Detection from Multimedia