Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

A Systematic Literature Review of Recognition of Compound Facial Expression of Emotions

A survey on facial emotion recognition techniques: A state-of-the-art literature review

A Survey on Facial Expression Recognition of Static and Dynamic Emotions

Compound facial expressions of emotion

Emotion Recognition Using Different Sensors, Emotion Models, Methods and Datasets: A Comprehensive Review

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Compound facial expressions recognition approach using DCGAN and CNN

Emotion in a Century: A Review of Emotion Recognition

Facial expression and body gesture emotion recognition: A systematic review on the use of visual data in affective computing

A survey on analysis of human faces and facial expressions datasets

[Infections likely to develop in immunocompromised hosts: Pneumocystis carinii pneumonia].

Facial expression recognition (FER) survey: a vision, architectural elements, and future directions

Facial emotion recognition: A comprehensive review

A Survey of AI-Based Facial Emotion Recognition: Features, ML & DL Techniques, Age-Wise Datasets and Future Directions

A Survey of the Trends in Facial and Expression Recognition Databases and Methods

Face Emotion Recognition Based on Machine Learning: A Review

Speech Emotion Recognition Systems: A Comprehensive Review on Different Methodologies

A Comparison of Methods of Facial Expression Recognition

A Comprehensive Review of Speech Emotion Recognition Systems

A Survey on Facial Emotion Recognition and Fake Emotion Detection Techniques

Generative Technology for Human Emotion Recognition: A Scope Review