Abstract:Multi-modal emotion analysis, as an important direction in affective computing, has attracted increasing attention in recent years. Most existing multi-modal emotion recognition studies are targeted at a classification task that aims to assign a specific emotion category to a combination of several heterogeneous input data, including multimedia signals and physiological signals. Compared to single-class emotion recognition, a growing number of recent psychological evidence suggests that different discrete emotions may co-exist at the same time, which promotes the development of mixed-emotion recognition to identify a mixture of basic emotions. Although most current studies treat it as a multi-label classification task, in this work, we focus on a challenging situation where both positive and negative emotions are presented simultaneously, and propose a multi-modal mixed emotion recognition framework, namely EmotionDict. The key characteristics of our EmotionDict include the following. (1) Inspired by the psychological evidence that such a mixed state can be represented by combinations of basic emotions, we address mixed emotion recognition as a label distribution learning task. An emotion dictionary has been designed to disentangle the mixed emotion representations into a weighted sum of a set of basic emotion elements in a shared latent space and their corresponding weights. (2) While many existing emotion distribution studies are built on a single type of multimedia signal (such as text, image, audio, and video), we incorporate physiological and overt behavioral multi-modal signals, including electroencephalogram (EEG), peripheral physiological signals, and facial videos, which directly display the subjective emotions. These modalities have diverse characteristics given that they are related to the central or peripheral nervous system, and the motor cortex. (3) We further design auxiliary tasks to learn modality attentions for modality integration. Experiments on two datasets show that our method outperforms existing state-of-the-art approaches on mixed-emotion recognition.

Mixed Emotion Recognition and Priming Effect on Cognitive Control

Mixed Emotion Recognition Based on EEG Signals

Temporal Convolutional Network-Enhanced Real-Time Implicit Emotion Recognition with an Innovative Wearable fNIRS-EEG Dual-Modal System

Multimodal Emotion Classification Method and Analysis of Brain Functional Connectivity Networks

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Multimodal Physiological Analysis of Impact of Emotion on Cognitive Control in VR

Emotion Dictionary Learning with Modality Attentions for Mixed Emotion Exploration

Temporal aware Mixed Attention-based Convolution and Transformer Network for cross-subject EEG emotion recognition

Design and Implementation of an Emotion Analysis System Based on EEG Signals

Neurofeedback Training with an Electroencephalogram-based Brain-Computer Interface Enhances Emotion Regulation

A Comprehensive Interaction in Multiscale Multichannel EEG Signals for Emotion Recognition

A Fine-Grained Approach for EEG-Based Emotion Recognition Using Clustering and Hybrid Deep Neural Networks

Recognition of Emotions Using Multichannel EEG Data and DBN-GC-Based Ensemble Deep Learning Framework

Naturalistic Emotion Recognition Using EEG and Eye Movements

Multi-Source EEG Emotion Recognition via Dynamic Contrastive Domain Adaptation

Progressive Graph Convolution Network for EEG Emotion Recognition

Enhancing Emotion Detection with Non-invasive Multi-Channel EEG and Hybrid Deep Learning Architecture

Emotion Recognition Based on Brain Connectivity Reservoir and Valence Lateralization for Cyber-Physical-Social Systems

Electroencephalogram Access for Emotion Recognition Based on a Deep Hybrid Network

Interaction Between Dynamic Affection and Arithmetic Cognitive Ability: A Practical Investigation with EEG Measurement