Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

Text-Based Emotion Recognition Using Deep Learning

Text-Based Emotion Recognition Using Deep Learning Approach

Deep learning approach to text analysis for human emotion detection from big data

Design of Hybrid Deep Learning Model for Text-based Emotion Recognition

Emotion Detection on text using Machine Learning and Deep Learning Techniques

Speech Emotion Recognition Using Deep Learning

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

A Survey of Textual Emotion Recognition and Its Challenges

Deep learning for affective computing: Text-based emotion recognition in decision support

Emotion Classification in Short English Texts using Deep Learning Techniques

Deep Learning based Emotion Recognition System Using Speech Features and Transcriptions

Optimal Facial Feature Based Emotional Recognition Using Deep Learning Algorithm

A Sentiment-and-Semantics-Based Approach for Emotion Detection in Textual Conversations

REDAffectiveLM: Leveraging Affect Enriched Embedding and Transformer-based Neural Language Model for Readers' Emotion Detection

Deep Learning Techniques for Speech Emotion Recognition: A Review

An adaptive search optimizer-based deep Bi-LSTM for emotion recognition using electroencephalogram signal

A review on emotion detection by using deep learning techniques

Facial Emotions Recognition Using Deep Learning Technology

DeepEmotex: Classifying Emotion in Text Messages using Deep Transfer Learning

Deep Broad Learning for Emotion Classification in Textual Conversations

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues