Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

Interpretable Deep Neural Networks for Dimensional and Categorical Emotion Recognition in-the-wild

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Multi-view Domain-Adaptive Representation Learning for EEG-based Emotion Recognition

A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild

Emotion Recognition in Audio and Video Using Deep Neural Networks

End-to-End Multimodal Emotion Recognition using Deep Neural Networks

Exploring Discriminative Representations for Image Emotion Recognition with CNNs.

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

Human Emotion Recognition with Electroencephalographic Multidimensional Features by Hybrid Deep Neural Networks

Multimodal Emotion Recognition From EEG Signals and Facial Expressions

A Dimensional Emotion Analysis Algorithm Based on Feature Reuse Mechanism

Deep learning-based dimensional emotion recognition combining the attention mechanism and global second-order feature representations

The application of deep learning in computer vision: Facial emotion recognition based on convolutional neural network

Leveraging Valence and Activation Information Via Multi-Task Learning for Categorical Emotion Recognition.

An improved multi-input deep convolutional neural network for automatic emotion recognition

Emotion Recognition from Multi-Channel Eeg Data Through Convolutional Recurrent Neural Network

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

A Combined CNN Architecture for Speech Emotion Recognition

A Systematic Exploration Of Deep Neural Networks For Eda-Based Emotion Recognition

A deep interpretable representation learning method for speech emotion recognition