Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

Quantified Facial Temporal-Expressiveness Dynamics for Affect Analysis

Quantified Facial Expressiveness for Affective Behavior Analytics

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Implicit Media Tagging and Affect Prediction from video of spontaneous facial expressions, recorded with depth camera

Modeling Dynamics of Facial Behavior for Mental Health Assessment

Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space

4DME: A Spontaneous 4D Micro-Expression Dataset with Multimodalities

Quantifying human sensitivity to spatio-temporal information in dynamic faces

Sensing emotional valence and arousal dynamics through automated facial action unit analysis

3D/4D Facial Expression Analysis: an Advanced Annotated Face Model Approach

Beyond Facial Expressions: Learning Human Emotion from Body Gestures.

Context-Dependent Models for Predicting and Characterizing Facial Expressiveness

Temporal Enhancement for Video Affective Content Analysis

Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features

Automatic facial expression recognition for affective computing based on bag of distances.

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

A Deep Quantum Convolutional Neural Network Based Facial Expression Recognition For Mental Health Analysis

Exploring Facial Biomarkers for Depression through Temporal Analysis of Action Units

A Unified Approach to Facial Affect Analysis: the MAE-Face Visual Representation.

An Ensemble Approach for Facial Expression Analysis in Video