Abstract:With the release of increasing open-source emotion recognition datasets on social media platforms and the rapid development of computing resources, multimodal emotion recognition tasks (MER) have begun to receive widespread research attention. The MER task extracts and fuses complementary semantic information from different modalities, which can classify the speaker's emotions. However, the existing feature fusion methods have usually mapped the features of different modalities into the same feature space for information fusion, which can not eliminate the heterogeneity between different modalities. Therefore, it is challenging to make the subsequent emotion class boundary learning. To tackle the above problems, we have proposed a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation, which can achieve information interaction between modalities and eliminate heterogeneity among modalities. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Specifically, we construct a graph structure for three modal features and perform contrastive representation learning on nodes with different emotions in the same modality and the same emotion in different modalities, which can improve the feature representation ability of nodes. Extensive experimental works show that the ARL-IIGCN method can significantly improve emotion recognition accuracy on IEMOCAP and MELD datasets.

Cross-Culture Multimodal Emotion Recognition With Adversarial Learning

Adversarial Domain Adaption for Multi-Cultural Dimensional Emotion Recognition in Dyadic Interactions

Emotion Recognition in Videos via Fusing Multimodal Features.

Capturing Emotion Distribution for Multimedia Emotion Tagging

Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition

Hybrid Multi-Task Learning for End-To-End Multimodal Emotion Recognition

Learning Robust Multi-Modal Representation for Multi-Label Emotion Recognition Via Adversarial Masking and Perturbation

Cross lingual speech emotion recognition via triple attentive asymmetric convolutional neural network

Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition

A Versatile Multimodal Learning Framework For Zero-shot Emotion Recognition

Multi-View Common Space Learning For Emotion Recognition In The Wild

A Cross-Culture Study on Multimodal Emotion Recognition Using Deep Learning

Cross-Corpus Speech Emotion Recognition Based on Multi-Task Learning and Subdomain Adaptation

Context-Dependent Domain Adversarial Neural Network for Multimodal Emotion Recognition

Multi-Subdomain Adversarial Network for Cross-Subject EEG-based Emotion Recognition

Improving Cross-Corpus Speech Emotion Recognition with Adversarial Discriminative Domain Generalization (ADDoG)

Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion Recognition

Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversations

Emo-DNA: Emotion Decoupling and Alignment Learning for Cross-Corpus Speech Emotion Recognition