Abstract:In the last few years, the multi-modal emotion recognition has become an important research issue in the affective computing community due to its wide range of applications that include mental disease diagnosis, human behavior understanding, human machine/robot interaction or autonomous driving systems. In this paper, we introduce a novel end-to-end multimodal emotion recognition methodology, based on audio and visual fusion designed to leverage the mutually complementary nature of features while maintaining the modality-specific information. The proposed method integrates spatial, channel and temporal attention mechanisms into a visual 3D convolutional neural network (3D-CNN) and temporal attention into an audio 2D convolutional neural network (2D-CNN) to capture the intra-modal features characteristics. Further, the inter-modal information is captured with the help of an audio-video (A-V) cross-attention fusion technique that effectively identifies salient relationships across the two modalities. Finally, by considering the semantic relations between the emotion categories, we design a novel classification loss based on an emotional metric constraint that guides the attention generation mechanisms. We demonstrate that by exploiting the relations between the emotion categories our method yields more discriminative embeddings, with more compact intra-class representations and increased inter-class separability. The experimental evaluation carried out on the RAVDESS ( The Ryerson Audio-Visual Database of Emotional Speech and Song ), and CREMA-D ( Crowd-sourced Emotional Multimodal Actors Dataset ) datasets validates the proposed methodology, which leads to average accuracy scores of 89.25% and 84.57%, respectively. In addition, when compared to state-of-the-art techniques, the proposed solution shows superior performances, with gains in accuracy ranging in the [1.72%, 11.25%] interval.

Multimodal Emotion Recognition Using Different Fusion Techniques

Emotion Recognition in Videos via Fusing Multimodal Features.

Multimodal Emotion Recognition by Combining Physiological Signals and Facial Expressions: a Preliminary Study.

Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition

Multimodal fusion framework: A multiresolution approach for emotion classification and recognition from physiological signals

Multimodal emotion recognition model via hybrid model with improved feature level fusion on facial and EEG feature set

Multimodal Emotion Recognition Framework Using a Decision-Level Fusion and Feature-Level Fusion Approach

Multimodal Emotion Recognition Based on Feature Fusion.

Multimodal Emotion Recognition System Using Machine Learning Classifier

Multi-Modal Emotion Recognition Feature Extraction and Data Fusion Methods Evaluation

Multimodal modelling of human emotion using sound, image and text fusion

Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

Multimodal Emotion Recognition based on the Fusion of EEG Signals and Eye Movement Data

Feature-level fusion of multimodal physiological signals for emotion recognition

Multimodal Emotion Recognition Based on Feature Selection and Extreme Learning Machine in Video Clips.

Emotion recognition framework using multiple modalities for an effective human–computer interaction

Interpretable Multimodal Emotion Recognition using Hybrid Fusion of Speech and Image Data

Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics

Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features

Multi-modal Emotion Recognition Based on Speech and Image.

Multimodal Emotion Recognition Using Heterogeneous Ensemble Techniques