Abstract:Multi-modal emotion analysis, as an important direction in affective computing, has attracted increasing attention in recent years. Most existing multi-modal emotion recognition studies are targeted at a classification task that aims to assign a specific emotion category to a combination of several heterogeneous input data, including multimedia signals and physiological signals. Compared to single-class emotion recognition, a growing number of recent psychological evidence suggests that different discrete emotions may co-exist at the same time, which promotes the development of mixed-emotion recognition to identify a mixture of basic emotions. Although most current studies treat it as a multi-label classification task, in this work, we focus on a challenging situation where both positive and negative emotions are presented simultaneously, and propose a multi-modal mixed emotion recognition framework, namely EmotionDict. The key characteristics of our EmotionDict include the following. (1) Inspired by the psychological evidence that such a mixed state can be represented by combinations of basic emotions, we address mixed emotion recognition as a label distribution learning task. An emotion dictionary has been designed to disentangle the mixed emotion representations into a weighted sum of a set of basic emotion elements in a shared latent space and their corresponding weights. (2) While many existing emotion distribution studies are built on a single type of multimedia signal (such as text, image, audio, and video), we incorporate physiological and overt behavioral multi-modal signals, including electroencephalogram (EEG), peripheral physiological signals, and facial videos, which directly display the subjective emotions. These modalities have diverse characteristics given that they are related to the central or peripheral nervous system, and the motor cortex. (3) We further design auxiliary tasks to learn modality attentions for modality integration. Experiments on two datasets show that our method outperforms existing state-of-the-art approaches on mixed-emotion recognition.

Reader's emotion prediction based on weighted latent Dirichlet allocation and multi-label k-nearest neighbor model

Emotion Detection in Online Social Networks: A Multilabel Learning Approach

Attention-Based BiLSTM Network with Lexical Feature for Emotion Classification

Predicting Social Emotions from Readers’ Perspective

A multi-modal and multi-scale emotion-enhanced inference model based on fuzzy recognition

Two-View Label Propagation to Semi-supervised Reader Emotion Classification.

Chinese Language Literature Emotion Analysis Model Based on Unbalanced K-Nearest Neighbor Classification Method

Prediction of emotion distribution of images based on weighted K-nearest neighbor-attention mechanism

Emotion classification for short texts: an improved multi-label method

REDAffectiveLM: Leveraging Affect Enriched Embedding and Transformer-based Neural Language Model for Readers' Emotion Detection

MODELLING A NOVEL APPROACH FOR EMOTION RECOGNITION USING LEARNING AND NATURAL LANGUAGE PROCESSING

A LDA-based approach to lyric emotion regression

Mining Social Emotions from Affective Text

Emotion Dictionary Learning with Modality Attentions for Mixed Emotion Exploration

An Empirical Study of Emotion Analysis with Different Distributed Representation Methods for Chinese Microblogs

Emotion-enhanced classification based on fuzzy reasoning

Emotion Detection In Online Social Network Based On Multi-Label Learning

Microblog User Emotion Analysis Method Based on Improved Hierarchical Attention Mechanism and BiLSTM

Emotion recognition of social media users based on deep learning

From Independent Prediction to Re-ordered Prediction: Integrating Relative Position and Global Label Information to Emotion Cause Identification