Abstract:Emotion analysis, an integral aspect of human–machine interactions, has witnessed significant advancements in recent years. With the rise of multimodal data sources such as speech, text, and images, there is a profound need for a comprehensive review of pivotal elements within this domain. Our paper delves deep into the realm of emotion analysis, examining multimodal data sources encompassing speech, text, images, and physiological signals. We provide a curated overview of relevant literature, academic forums, and competitions. Emphasis is laid on dissecting unimodal processing methods, including preprocessing, feature extraction, and tools across speech, text, images, and physiological signals. We further discuss the nuances of multimodal data fusion techniques, spotlighting early, late, model, and hybrid fusion strategies. Key findings indicate the essentiality of analyzing emotions across multiple modalities. Detailed discussions on emotion elicitation, expression, and representation models are presented. Moreover, we uncover challenges such as dataset creation, modality synchronization, model efficiency, limited data scenarios, cross-domain applicability, and the handling of missing modalities. Practical solutions and suggestions are provided to address these challenges. The realm of multimodal emotion analysis is vast, with numerous applications ranging from driver sentiment detection to medical evaluations. Our comprehensive review serves as a valuable resource for both scholars and industry professionals. It not only sheds light on the current state of research but also highlights potential directions for future innovations. The insights garnered from this paper are expected to pave the way for subsequent advancements in deep multimodal emotion analysis tailored for real-world deployments.

A Review of Multimodal Interaction

Multimodal interaction: A review

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

A review on multimodal interaction in Mixed Reality Environment

Multimodal Systems: Taxonomy, Methods, and Challenges

Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review

A review of affective computing: From unimodal analysis to multimodal fusion

Recent advancements in multimodal human-robot interaction

A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Review of Non-Intrusive Interaction Based on Camera

Progress and Prospects of Multimodal Fusion Methods in Physical Human–Robot Interaction: A Review

Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

Introduction: Multimodal interaction

Review on Perceptual User Interfaces Based on Motion Sensors

Data fusion methods in multimodal human computer dialog.

Cross-modal Communication Technology: A Survey

Recent Advances and Trends in Multimodal Deep Learning: A Review

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal human–computer interaction in interventional radiology and surgery: a systematic literature review