Abstract:In the last few years, the multi-modal emotion recognition has become an important research issue in the affective computing community due to its wide range of applications that include mental disease diagnosis, human behavior understanding, human machine/robot interaction or autonomous driving systems. In this paper, we introduce a novel end-to-end multimodal emotion recognition methodology, based on audio and visual fusion designed to leverage the mutually complementary nature of features while maintaining the modality-specific information. The proposed method integrates spatial, channel and temporal attention mechanisms into a visual 3D convolutional neural network (3D-CNN) and temporal attention into an audio 2D convolutional neural network (2D-CNN) to capture the intra-modal features characteristics. Further, the inter-modal information is captured with the help of an audio-video (A-V) cross-attention fusion technique that effectively identifies salient relationships across the two modalities. Finally, by considering the semantic relations between the emotion categories, we design a novel classification loss based on an emotional metric constraint that guides the attention generation mechanisms. We demonstrate that by exploiting the relations between the emotion categories our method yields more discriminative embeddings, with more compact intra-class representations and increased inter-class separability. The experimental evaluation carried out on the RAVDESS ( The Ryerson Audio-Visual Database of Emotional Speech and Song ), and CREMA-D ( Crowd-sourced Emotional Multimodal Actors Dataset ) datasets validates the proposed methodology, which leads to average accuracy scores of 89.25% and 84.57%, respectively. In addition, when compared to state-of-the-art techniques, the proposed solution shows superior performances, with gains in accuracy ranging in the [1.72%, 11.25%] interval.

AuxFormer: Robust Approach to Audiovisual Emotion Recognition

Robust Audiovisual Emotion Recognition: Aligning Modalities, Capturing Temporal Information, and Handling Missing Features

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Audio-Visual Fusion Network Based on Conformer for Multimodal Emotion Recognition

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

Multimodal Transformer Fusion for Continuous Emotion Recognition

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

Multimodal Emotion Recognition using Audio-Video Transformer Fusion with Cross Attention

A Cross-Modal Fusion Network Based on Self-Attention and Residual Structure for Multimodal Emotion Recognition

Emotion Recognition with Multimodal Transformer Fusion Framework Based on Acoustic and Lexical Information

Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learning

Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition

Multimodal emotion recognition based on a fusion of audiovisual information with temporal dynamics

Modality-collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition

Versatile audio-visual learning for emotion recognition

MATF: main-auxiliary transformer fusion for multi-modal sentiment analysis

Multimodal transformer augmented fusion for speech emotion recognition

Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion