Abstract:With the advancement of multimedia and human-computer interaction, it has become increasingly crucial to perceive people's emotional states in dynamic data (e.g., video, audio, text stream) in order to effectively serve them. Emotion recognition has emerged as a prominent research area over the past decades. Traditional methods for emotion recognition heavily rely on manually crafted features and primarily focus on uni-modality. However, these approaches encounter challenges in extracting sufficient discriminative information for complex emotion recognition tasks. To tackle this issue, deep neural model-based methods have gained significant popularity in emotion recognition tasks. These methods leverage deep neural models to automatically learn more discriminative emotional features, thereby addressing the problem of poor discriminability associated with manually designed features. Moreover, deep neural models are also employed to integrate information across multiple modalities, thereby enhancing the extraction of discriminative information. In this paper, we provide a comprehensive review of the relevant studies on deep neural model-based emotion recognition in dynamic data using facial, speech, and textual cues published within the past five years. Specifically, we first explain discretized and continuous representations of emotions by introducing widely accepted emotion models. Subsequently, we elucidate how advanced methods integrate different neural models by scoping these methods using variant popular deep neural models (e.g. Transformer), along with corresponding preprocessing mechanisms. In addition, we present the development trend by surveying diverse datasets, metrics, and competitive performances. Finally, we have a discussion and explore significant research challenges and opportunities. Our survey bridges the gaps in the literature since existing surveys are narrow in focus, either exclusively covering single-modal methods, solely concentrating on multi-modal methods, overlooking certain aspects of face, speech, and text, or emphasizing outdated methodologies.

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Multi-modal emotion analysis from facial expressions and electroencephalogram.

Survey of deep emotion recognition in dynamic data using facial, speech and textual cues

Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Exploring contactless techniques in multimodal emotion recognition: insights into diverse applications, challenges, solutions, and prospects

Survey on AI-Based Multimodal Methods for Emotion Detection

Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG

A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition

Multimodal Emotion Recognition by Extracting Common and Modality-Specific Information.

Multimodal Emotion Recognition Based on Feature Fusion.

A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Multimodal Emotion Recognition Based on Facial Expressions, Speech, and Body Gestures

Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review

A multimodal emotion recognition model integrating speech, video and MoCAP

Realization of Self-Adaptive Higher Teaching Management Based Upon Expression and Speech Multimodal Emotion Recognition

A Novel Emotion-Aware Method Based on the Fusion of Textual Description of Speech, Body Movements, and Facial Expressions

Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection