Abstract:Over the last decade, automatic emotion recognition has become well established. The gold standard target is thereby usually calculated based on multiple annotations from different raters. All related efforts assume that the emotional state of a human subject can be identified by a 'hard' category or a unique value. This assumption tries to ease the human observer's subjectivity when observing patterns such as the emotional state of others. However, as the number of annotators cannot be infinite, uncertainty remains in the emotion target even if calculated from several, yet few human annotators. The common procedure to use this same emotion target in the learning process thus inevitably introduces noise in terms of an uncertain learning target. In this light, we propose a 'soft' prediction framework to provide a more human-like and comprehensive prediction of emotion. In our novel framework, we provide an additional target to indicate the uncertainty of human perception based on the inter-rater disagreement level, in contrast to the traditional framework which is merely producing one single prediction (category or value). To exploit the dependency between the emotional state and the newly introduced perception uncertainty, we implement a multi-task learning strategy. To evaluate the feasibility and effectiveness of the proposed soft prediction framework, we perform extensive experiments on a time- and value-continuous spontaneous audiovisual emotion database including late fusion results. We show that the soft prediction framework with multi-task learning of the emotional state and its perception uncertainty significantly outperforms the individual tasks in both the arousal and valence dimensions.

Exploring Perception Uncertainty for Emotion Recognition in Dyadic Conversation and Music Listening

From Hard to Soft: Towards More Human-like Emotion Recognition by Modelling the Perception Uncertainty

A Efficient Multimodal Framework for Large Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition

Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition.

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Multi-Modal Multi-Cultural Dimensional Continues Emotion Recognition In Dyadic Interactions

Prediction-based Learning for Continuous Emotion Recognition in Speech

Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions.

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

Inconsistency-Based Multi-Task Cooperative Learning for Emotion Recognition

Speech Emotion Recognition in Dyadic Dialogues with Attentive Interaction Modeling

Towards A Robust Group-level Emotion Recognition via Uncertainty-Aware Learning

EmotionIC: emotional inertia and contagion-driven dependency modeling for emotion recognition in conversation

Exploration meets exploitation: Multitask learning for emotion recognition based on discrete and dimensional models

Joint user-interest and social-influence emotion prediction for individuals

A Deep Bidirectional Long Short-Term Memory Based Multi-Scale Approach for Music Dynamic Emotion Prediction