Abstract:Emotion detection, hence, is an important step toward making human-computer interaction a more enhanced process, where systems are made capable of identifying and responding to the emotional state of users. In fact, multimodal emotion detection systems in which both auditory and visual information are fused are emerging, and these approaches toward expressive emotional states are complementary and robust. Multimodal systems enhance the quality of interacting and, through many applications, can diagnose emotional disorders, monitor automotive safety, and improve human-robot interactions. In nature, the high-dimensional space and dynamic threats have resulted in obtaining low accuracy and high computational cost in applying the traditional models based on single-modality data. On the other hand, multimodal systems explore the synergy between audio and visual data, giving better performance and higher accuracy in inferring subtle emotional expressions. The latest improvement was done on these systems using recent advancements in transfer learning and deep learning techniques.That being said, this research Proposal devises a multimodal emotion recognition system integrating speech and face information through transfer learning for improved accuracy and robustness. Serving this purpose, the objectives of this research entail the effective comparison among different transfer-learning strategies, including the impact of pre-trained models in speech-based emotion recognition, and to introduce the role of voice activity detection in the process. Advanced neural network architectures like Spatial Transformer Networks and bidirectional LSTM in facial emotion recognition will also be tested. Early and late fusion strategies will also be used to find the best strategy for combining speech and facial data.This research will target several challenges that involve the complexity of data, balancing of the model performance-robustness balance, computational limitations, and standardization of evaluations in developing a working and robust emotion recognition system to enhance digital interaction and apply in practical areas. The goal is to create a system that oversteps the limitation of single-modality models through state-of-the-art advances in deep learning, as well as front-line improvements in transfer learning, in the manner of emotion detection performance.

Facial Emotion Recognition with Inter-Modality-Attention-Transformer-Based Self-Supervised Learning

Emotion Recognition in Videos via Fusing Multimodal Features.

Multimodal Emotion Recognition With Transformer-Based Self Supervised Feature Fusion

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

Multimodal Emotion Recognition Based on Deep Temporal Features Using Cross-Modal Transformer and Self-Attention

Multimodal Transformer with Learnable Frontend and Self Attention for Emotion Recognition

Multimodal emotion recognition from facial expression and speech based on feature fusion

Multimodal Transformer Fusion for Continuous Emotion Recognition

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

Multimodal Transformer Fusion for Emotion Recognition: A Survey

Multimodal transformer augmented fusion for speech emotion recognition

Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals

Enhancing Emotion Recognition through Multimodal Systems and Advanced Deep Learning Techniques

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Multimodal Multi-View Spectral-Spatial-Temporal Masked Autoencoder for Self-Supervised Emotion Recognition

Multimodal Emotion Recognition Using Different Fusion Techniques

Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Learning Modality-Fused Representation Based on Transformer for Emotion Analysis

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations