Abstract:Emotion computing is a necessary part of advanced human–computer interaction. An appropriate description of a character's facial expressions, body languages, and speaking styles in novels always enables readers to infer the character's emotions. Moreover, multimodal information is complementary and integrated. Fusing the information from multiple modes into a textual modal can get better fusion results and overcome the bias of understanding the unimodal information. Inspired by these facts, we develop a novel emotion-aware method by the fusion of textual description of speech, body movements, and facial expression, which reduces the dimensionality of speech, body movements, and facial expressions by unifying three types of information into a unified component. Specifically, to fuse multimodel features for emotion recognition, we propose a two-stage neural network. First, bidirectional long short-term memory-conditional random fields (Bi-LSTM-CRF) and back-propagation neural network (BPNN) are used to analyze the extracted vocal and visual features of facial expressions, body movements, and speeches, which aims to obtain textual descriptions of different features. Second, the textual descriptions of the features are fused through a neural network with a self-organization map (SOM) layer and are used to compensate layers that are trained by web-based corpus. The advantages of this method are to utilize depth information to track facial and bodily movement, and employ an explainable textual intermediate representation to fuse the features. We experimentally tested the emotion-aware system in real-world applications, and the results indicate that our system can quickly and steadily recognize human emotions. Compared with other unimodal and multimodal-fusion algorithms, our method is more precise, which can improve the accuracy by up to 30% compared with the unimodal method.

A Novel Active-Learning Based Emotion-Vision-Transformer Network for Expression Recognition

Cgan Based Facial Expression Recognition for Human-Robot Interaction

Adaptively Learning Facial Expression Representation via C-F Labels and Distillation

EmotioNet Challenge: Recognition of facial expressions of emotion in the wild

Expression Recognition Based on Human Emotion-Attention Circuit

Emotion Recognition for Challenged People Facial Appearance in Social using Neural Network

Expression Analysis Based on Face Regions in Real-world Conditions.

Expression Analysis Based on Face Regions in Read-world Conditions

2D/3D Expression Generation Using Advanced Learning Techniques and the Emotion Wheel

Building and validation of a set of facial expression images to detect emotions: a transcultural study

Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network

Personalized Broad Learning System for Facial Expression

Deep Learning Based Emotion Recognition and Visualization of Figural Representation

Implementation of an adaptive E-learning platform with facial emotion recognition

An Ensemble Approach for Facial Expression Analysis in Video

Face Emotion Recognization Using Dataset Augmentation Based on Neural Network

A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study

Emotion Recognition Using Transformers with Masked Learning

Facial Emotion Recognition Using a Novel Fusion of Convolutional Neural Network and Local Binary Pattern in Crime Investigation

Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait