Abstract:Emotion computing is a necessary part of advanced human–computer interaction. An appropriate description of a character's facial expressions, body languages, and speaking styles in novels always enables readers to infer the character's emotions. Moreover, multimodal information is complementary and integrated. Fusing the information from multiple modes into a textual modal can get better fusion results and overcome the bias of understanding the unimodal information. Inspired by these facts, we develop a novel emotion-aware method by the fusion of textual description of speech, body movements, and facial expression, which reduces the dimensionality of speech, body movements, and facial expressions by unifying three types of information into a unified component. Specifically, to fuse multimodel features for emotion recognition, we propose a two-stage neural network. First, bidirectional long short-term memory-conditional random fields (Bi-LSTM-CRF) and back-propagation neural network (BPNN) are used to analyze the extracted vocal and visual features of facial expressions, body movements, and speeches, which aims to obtain textual descriptions of different features. Second, the textual descriptions of the features are fused through a neural network with a self-organization map (SOM) layer and are used to compensate layers that are trained by web-based corpus. The advantages of this method are to utilize depth information to track facial and bodily movement, and employ an explainable textual intermediate representation to fuse the features. We experimentally tested the emotion-aware system in real-world applications, and the results indicate that our system can quickly and steadily recognize human emotions. Compared with other unimodal and multimodal-fusion algorithms, our method is more precise, which can improve the accuracy by up to 30% compared with the unimodal method.

Emotion Evaluation Through Body Movements Based on Silhouette Extraction.

Emotional Classification and Prediction of Body Movements Based on Silhouette

Emotion Detection through Body Gesture and Face

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Emotion Recognition via Environmental Context and Human Body

Estimating Emotional Intensity from Body Poses for Human-Robot Interaction

ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild

Bi-modal Emotion Recognition Based on Vision-CSI

Bodily expressed emotion understanding through integrating Laban movement analysis

Real-time Automatic Emotion Recognition from Body Gestures

Emotion Expression in Human Body Posture and Movement: A Survey on Intelligible Motion Factors, Quantification and Validation

A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

Happy Index

Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation

Recognition of Emotions Using Kinects

Human emotion and cognition recognition from body language of the head using soft computing techniques

Happy-sad expression recognition using emotion geometry feature and support vector machine

Happy Index - Analysis Based on Automatic Recognition of Emotion Flow.

Recognizing Emotions Based on Human Actions in Videos.

Bi-Model Engagement Emotion Recognition Based on Facial and Upper-Body Landmarks and Machine Learning Approaches

Emotion Recognition Using Kinect Motion Capture Data of Human Gaits