Abstract:Emotions are central for human intelligence and should have a similar role in AI. When it comes to emotion recognition, however, analysis cues for robots were mostly limited to human facial expressions and speech. As an alternative important non-verbal communicative fashion, the body gesture is proved to be capable of conveying emotional information which should gain more attention. Inspired by recent researches on micro-expressions, in this paper, we try to explore a specific group of gestures which are spontaneously and unconsciously elicited by inner feelings. These gestures are different from common gestures for facilitating communications or to express feelings on ones own initiative and always ignored in our daily life. This kind of subtle body movements is known as ‘micro-gestures’ (MGs). Work of interpreting the human hidden emotions via these specific gestural behaviors in unconstrained situations, however, is limited. It is because of an unclear correspondence between body movements and emotional states which need multidisciplinary efforts from computer science, psychology, and statistic researchers. To fill the gap, we built a novel Spontaneous Micro-Gesture (SMG) dataset containing 3,692 manually labeled gesture clips. The data collection from 40 participants was conducted through a story-telling game with two emotional state settings. In this paper, we explored the emotional gestures with a sign-based measurement. To verify the latent relationship between emotional states and MGs, we proposed a framework that encodes the objective gestures to a Bayesian network to infer the subjective emotional states. Our experimental results revealed that, most of the participants would do ‘micro-gestures’ spontaneously to relieve their mental strains. We also carried out a human test on ordinary and trained people for comparison. The performance of both our framework and human beings was evaluated on 142 testing instances (71 for each emotional state) by subject-independent testing. To authors’ best knowledge, this is the first presented MG dataset. Results showed that the proposed MG recognition method achieved promising performance. We also showed that MGs could be helpful cues for the recognition of hidden emotional states.

MuSE: a Multimodal Dataset of Stressed Emotion

The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress

MUSER: MUltimodal Stress Detection using Emotion Recognition as an Auxiliary Task

The MuSe 2022 Multimodal Sentiment Analysis Challenge: Humor, Emotional Reactions, and Stress

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Multimodal Sentiment Analysis Based on Recurrent Neural Network and Multimodal Attention.

The MuSe 2023 Multimodal Sentiment Analysis Challenge: Mimicked Emotions, Cross-Cultural Humour, and Personalisation

Hybrid Multimodal Feature Extraction, Mining and Fusion for Sentiment Analysis

A Multimodal Dataset for Mixed Emotion Recognition

SMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State Analysis.

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

Towards Multimodal Emotional Support Conversation Systems

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

EmpathicSchool: A multimodal dataset for real-time facial expressions and physiological data analysis under different stress conditions

Analyze Spontaneous Gestures for Emotional Stress State Recognition: A Micro-gesture Dataset and Analysis with Deep Learning

Multimodal Temporal Attention in Sentiment Analysis

MPED: A Multi-Modal Physiological Emotion Database for Discrete Emotion Recognition

M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model

The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition