Abstract:Emotion computing is a necessary part of advanced human–computer interaction. An appropriate description of a character's facial expressions, body languages, and speaking styles in novels always enables readers to infer the character's emotions. Moreover, multimodal information is complementary and integrated. Fusing the information from multiple modes into a textual modal can get better fusion results and overcome the bias of understanding the unimodal information. Inspired by these facts, we develop a novel emotion-aware method by the fusion of textual description of speech, body movements, and facial expression, which reduces the dimensionality of speech, body movements, and facial expressions by unifying three types of information into a unified component. Specifically, to fuse multimodel features for emotion recognition, we propose a two-stage neural network. First, bidirectional long short-term memory-conditional random fields (Bi-LSTM-CRF) and back-propagation neural network (BPNN) are used to analyze the extracted vocal and visual features of facial expressions, body movements, and speeches, which aims to obtain textual descriptions of different features. Second, the textual descriptions of the features are fused through a neural network with a self-organization map (SOM) layer and are used to compensate layers that are trained by web-based corpus. The advantages of this method are to utilize depth information to track facial and bodily movement, and employ an explainable textual intermediate representation to fuse the features. We experimentally tested the emotion-aware system in real-world applications, and the results indicate that our system can quickly and steadily recognize human emotions. Compared with other unimodal and multimodal-fusion algorithms, our method is more precise, which can improve the accuracy by up to 30% compared with the unimodal method.

Affect Detection from Semantic Interpretation of Drama Improvisation.

Affect detection from semantic and metaphorical interpretation of virtual drama

Text-driven Visual Prosody Generation for Embodied Conversational Agents

Emotion Detection from Speech to Enrich Multimedia Content

I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue

A Mood Semantic Awareness Model for Emotional Interactive Robots

Estimating human emotions using wording and sentence patterns

A Computational Approach to Emotion Recognition in Intelligent Agent

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

A Preliminary Research of Chinese Emotion Classification Model

Emotion Predicting Method Based on Emotion State Change of Personae According to the Other'S Utterances

Enhancing Emotional Induction in Virtual Reality: Innovative Fusion of Speech and Realistic Non-Verbal Emotional Expression

An Effective Ensemble Learning Framework for Affective Behaviour Analysis

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

A Semantic Analyzer For Aiding Emotion Recognition In Chinese

A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

Coupling Sentiment and Arousal Analysis Towards an Affective Dialogue Manager

Semi-Automatic Emotion Recognition From Chinese Text

Inferring User Emotive State Changes in Realistic Human-Computer Conversational Dialogs.

Research on Emotional Interaction and User Cognition

A Supplementary Feature Set for Sentiment Analysis in Japanese Dialogues