Abstract:Multi-modal emotion analysis, as an important direction in affective computing, has attracted increasing attention in recent years. Most existing multi-modal emotion recognition studies are targeted at a classification task that aims to assign a specific emotion category to a combination of several heterogeneous input data, including multimedia signals and physiological signals. Compared to single-class emotion recognition, a growing number of recent psychological evidence suggests that different discrete emotions may co-exist at the same time, which promotes the development of mixed-emotion recognition to identify a mixture of basic emotions. Although most current studies treat it as a multi-label classification task, in this work, we focus on a challenging situation where both positive and negative emotions are presented simultaneously, and propose a multi-modal mixed emotion recognition framework, namely EmotionDict. The key characteristics of our EmotionDict include the following. (1) Inspired by the psychological evidence that such a mixed state can be represented by combinations of basic emotions, we address mixed emotion recognition as a label distribution learning task. An emotion dictionary has been designed to disentangle the mixed emotion representations into a weighted sum of a set of basic emotion elements in a shared latent space and their corresponding weights. (2) While many existing emotion distribution studies are built on a single type of multimedia signal (such as text, image, audio, and video), we incorporate physiological and overt behavioral multi-modal signals, including electroencephalogram (EEG), peripheral physiological signals, and facial videos, which directly display the subjective emotions. These modalities have diverse characteristics given that they are related to the central or peripheral nervous system, and the motor cortex. (3) We further design auxiliary tasks to learn modality attentions for modality integration. Experiments on two datasets show that our method outperforms existing state-of-the-art approaches on mixed-emotion recognition.

Ensemble of Binary Classification for the Emotion Detection in Code-Switching Text.

A Multi-emotion Classification Method Based on BLSTM-MC in Code-Switching Text.

Ensemble of Neural Networks with Sentiment Words Translation for Code-Switching Emotion Detection.

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

Overview of NLPCC 2018 Shared Task 1: Emotion Detection in Code-Switching Text.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Emotion Detection in Code-switching Texts Via Bilingual and Sentimental Information.

A Bilingual Attention Network for Code-switched Emotion Prediction.

Emotion in Code-switching Texts: Corpus Construction and Analysis

Multi-view learning for emotion detection in code-switching texts

Leveraging Bilingual-View Parallel Translation for Code-Switched Emotion Detection with Adversarial Dual-Channel Encoder.

Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition

Attention-Based BiLSTM Network with Lexical Feature for Emotion Classification

Multi-modal Expression Recognition with Ensemble Method

TEII: Think, Explain, Interact and Iterate with Large Language Models to Solve Cross-lingual Emotion Detection

YNU-HPCC at WASSA 2023: Using Text-Mixed Data Augmentation for Emotion Classification on Code-Mixed Text Message.

Audio-Video Based Multimodal Emotion Recognition Using Svms and Deep Learning

A Parallel Two-Channel Emotion Classification Method for Chinese Text

Text Emotion Distribution Learning Via Multi-Task Convolutional Neural Network

Fine-Tuning BERT for Multi-Label Sentiment Analysis in Unbalanced Code-Switching Text.

Emotion Dictionary Learning with Modality Attentions for Mixed Emotion Exploration