Abstract:In this work, we focus on a special group of human body language -- the micro-gesture (MG), which differs from the range of ordinary illustrative gestures in that they are not intentional behaviors performed to convey information to others, but rather unintentional behaviors driven by inner feelings. This characteristic introduces two novel challenges regarding micro-gestures that are worth rethinking. The first is whether strategies designed for other action recognition are entirely applicable to micro-gestures. The second is whether micro-gestures, as supplementary data, can provide additional insights for emotional understanding. In recognizing micro-gestures, we explored various augmentation strategies that take into account the subtle spatial and brief temporal characteristics of micro-gestures, often accompanied by repetitiveness, to determine more suitable augmentation methods. Considering the significance of temporal domain information for micro-gestures, we introduce a simple and efficient plug-and-play spatiotemporal balancing fusion method. We not only studied our method on the considered micro-gesture dataset but also conducted experiments on mainstream action datasets. The results show that our approach performs well in micro-gesture recognition and on other datasets, achieving state-of-the-art performance compared to previous micro-gesture recognition methods. For emotional understanding based on micro-gestures, we construct complex emotional reasoning scenarios. Our evaluation, conducted with large language models, shows that micro-gestures play a significant and positive role in enhancing comprehensive emotional understanding. The scenarios we developed can be extended to other micro-gesture-based tasks such as deception detection and interviews. We confirm that our new insights contribute to advancing research in micro-gesture and emotional artificial intelligence.

Emotional gaze behavior generation in human-agent interaction.

Cgan Based Facial Expression Recognition for Human-Robot Interaction

Toward Automated Generation of Affective Gestures from Text:A Theory-Driven Approach

Study on Artificial Emotion and Its Application in the Behavior Selection Strategy of Agents

Artificial Emotion and Its Recognition, Modeling and Applications: an Overview

Multimodal emotion estimation and emotional synthesize for interaction virtual agent

Modeling of conversational agent with empathy mechanism

An Emotion-Driven Attention Model For Service Robot

Beyond Facial Expressions: Learning Human Emotion from Body Gestures.

Judging by the Look: The Impact of Robot Gaze Strategies on Human Cooperation

EMOTION: Expressive Motion Sequence Generation for Humanoid Robots with In-Context Learning

Implementation of Emotional Facial Actions for Human-Like Agents in a Multi-Tasking Environment

Context-Aware Head-and-Eye Motion Generation with Diffusion Model

Emotional Listener Portrait: Neural Listener Head Generation with Emotion

Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

Personality-Driven Gaze Animation with Conditional Generative Adversarial Networks

The design of expressive intelligent agent for human-computer interaction

Gaze estimation learning architecture as support to affective, social and cognitive studies in natural human-robot interaction

Interactive Generative Adversarial Networks for Facial Expression Generation in Dyadic Interactions

Identity-free Artificial Emotional Intelligence via Micro-Gesture Understanding

Vision-based Gaze Estimation: A Review