Abstract:Understanding emotion in context is a rising hotspot in the computer vision community. Existing methods lack reliable context semantics to mitigate uncertainty in expressing emotions and fail to model multiple context representations complementarily. To alleviate these issues, we present a context-aware emotion recognition framework that combines four complementary contexts. The first context is multimodal emotion recognition based on facial expression, facial landmarks, gesture and gait. Secondly, we adopt the channel and spatial attention modules to obtain the emotion semantics of the scene context. Inspired by sociology theory, we explore the emotion transmission between agents by constructing relationship graphs in the third context. Meanwhile, we propose a novel agent-object context, which aggregates emotion cues from the interactions between surrounding agents and objects in the scene to mitigate the ambiguity of prediction. Finally, we introduce an adaptive relevance fusion module for learning the shared representations among multiple contexts. Extensive experiments show that our approach outperforms the state-of-the-art methods on both EMOTIC and GroupWalk datasets. We also release a dataset annotated with diverse emotion labels, Human Emotion in Context (HECO). In practice, we compare with the existing methods on the HECO, and our approach obtains a higher classification average precision of 50.65% and a lower regression mean error rate of 0.7.

THU-HCSI at SemEval-2019 Task 3: Hierarchical Ensemble Classification of Contextual Emotion in Conversation.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

CAiRE_HKUST at SemEval-2019 Task 3: Hierarchical Attention for Dialogue Emotion Classification

THU_NGN at SemEval-2019 Task 3: Dialog Emotion Classification Using Attentional LSTM-CNN.

Attention-Enhanced Connectionist Temporal Classification for Discrete Speech Emotion Recognition

Samsung Research China-Beijing at SemEval-2024 Task 3: A multi-stage framework for Emotion-Cause Pair Extraction in Conversations

PKUSE at SemEval-2019 Task 3: Emotion Detection with Emotion-Oriented Neural Attention Network.

Emotion Recognition for Multiple Context Awareness.

Human Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder

Emotion Recognition via Environmental Context and Human Body

A Benchmark for Hierarchical Emotion Cause Extraction in Spoken Dialogues

On The Need Of Hierarchical Emotion Classification: Detecting The Implicit Feature Using Constrained Topic Model

Hierarchical speech emotion recognition using the valence-arousal model

Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023

SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations

MediaEval 2014: THU-HCSIL Approach to Emotion in Music Task Using Multi-level Regression.

HDEL: a hierarchical deep ensemble approach for text-based emotion detection

Hierarchical emotion-recognition framework based on discriminative brain neural network topology and ensemble co-decision strategy

Hierarchical emotion classification and emotion component analysis on chinese micro-blog posts

Classification of Emotional Speech Based on an Automatically Elaborated Hierarchical Classifier

JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models