Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

Emotion Analysis for the Upcoming Response in Open-Domain Human-Computer Conversation.

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction

Emotion Recognition via Environmental Context and Human Body

Neural Response Generation with Relevant Emotions for Short Text Conversation

Chinese Emotional Dialogue Response Generation via Reinforcement Learning

Emotional Human-Machine Conversation Generation Based on Long Short-Term Memory

Generating Empathetic Responses by Looking Ahead the User's Sentiment

Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation

Annotation and Detection of Emotion in Text-based Dialogue Systems with CNN

Emotion Eliciting Machine: Emotion Eliciting Conversation Generation based on Dual Generator

Integrating Natural Language Understanding and a Cognitive Approach to Textual Emotion Recognition for Generating Human-Like Responses

Automatically Select Emotion for Response via Personality-affected Emotion Transition

Emotion Analysis in Human–Robot Interaction

Emotion-and-knowledge grounded response generation in an open-domain dialogue setting

A State of the Art Technology with Machine Intelligence in Human Emotion Recognition

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

AHRNN: Attention‐Based Hybrid Robust Neural Network for Emotion Recognition

Inferring User Emotive State Changes in Realistic Human-Computer Conversational Dialogs.

What If Bots Feel Moods? Towards Controllable Retrieval-based Dialogue Systems with Emotion-Aware Transition Networks

I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue