Abstract:Emotions are central for human intelligence and should have a similar role in AI. When it comes to emotion recognition, however, analysis cues for robots were mostly limited to human facial expressions and speech. As an alternative important non-verbal communicative fashion, the body gesture is proved to be capable of conveying emotional information which should gain more attention. Inspired by recent researches on micro-expressions, in this paper, we try to explore a specific group of gestures which are spontaneously and unconsciously elicited by inner feelings. These gestures are different from common gestures for facilitating communications or to express feelings on ones own initiative and always ignored in our daily life. This kind of subtle body movements is known as ‘micro-gestures’ (MGs). Work of interpreting the human hidden emotions via these specific gestural behaviors in unconstrained situations, however, is limited. It is because of an unclear correspondence between body movements and emotional states which need multidisciplinary efforts from computer science, psychology, and statistic researchers. To fill the gap, we built a novel Spontaneous Micro-Gesture (SMG) dataset containing 3,692 manually labeled gesture clips. The data collection from 40 participants was conducted through a story-telling game with two emotional state settings. In this paper, we explored the emotional gestures with a sign-based measurement. To verify the latent relationship between emotional states and MGs, we proposed a framework that encodes the objective gestures to a Bayesian network to infer the subjective emotional states. Our experimental results revealed that, most of the participants would do ‘micro-gestures’ spontaneously to relieve their mental strains. We also carried out a human test on ordinary and trained people for comparison. The performance of both our framework and human beings was evaluated on 142 testing instances (71 for each emotional state) by subject-independent testing. To authors’ best knowledge, this is the first presented MG dataset. Results showed that the proposed MG recognition method achieved promising performance. We also showed that MGs could be helpful cues for the recognition of hidden emotional states.

Virtual Emotional Gestures to Assist in the Examination of the Mental Health of the Deaf-Mutes.

Hybrid Silent Speech Interface Through Fusion of Electroencephalography and Electromyography

Development of an Eog (Electro-Oculography) Based Human-Computer Interface

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

Facial Expression and EEG Fusion for Investigating Continuous Emotions of Deaf Subjects

A Multimodal Emotional Communication Based Humans-Robots Interaction System

Investigating of Deaf Emotion Cognition Pattern by EEG and Facial Expression Combination

A Virtual Keyboard Controlled By Eog

Building Language Communication between Deaf People and Hearing Society through Multimodal Human-Computer Interface

Analyze Spontaneous Gestures for Emotional Stress State Recognition: A Micro-gesture Dataset and Analysis with Deep Learning

A Method to Synthesize Chinese Sign Language Based on Virtual Human Technologies

Handtalker: A Multimodal Dialog System Using Sign Language and 3-D Virtual Human

Designing and Evaluating an Emotionally Responsive Virtual Patient Simulation

Exploring the Impact of Emotional Voice Integration in Sign-to-Speech Translators for Deaf-to-Hearing Communication

Inclusive Emotion Technologies: Addressing the Needs of d/Deaf and Hard of Hearing Learners in Video-Based Learning

Emotion Recognition of Subjects With Hearing Impairment Based on Fusion of Facial Expression and EEG Topographic Map

Remote Virtual Companion Via Tactile Codes and Voices for the People with Visual Impairment

Utilization of passive visual perception task in detecting patients with major depressive disorder for active health

Research on virtual entertainment robots based on machine learning algorithms providing psychological health services for college students

Design and Implementation of Gesture Recognition System Based on Flex Sensors

Development of a Sign Language Dialogue System for a Healing Dialogue Robot