Abstract:With the release of increasing open-source emotion recognition datasets on social media platforms and the rapid development of computing resources, multimodal emotion recognition tasks (MER) have begun to receive widespread research attention. The MER task extracts and fuses complementary semantic information from different modalities, which can classify the speaker's emotions. However, the existing feature fusion methods have usually mapped the features of different modalities into the same feature space for information fusion, which can not eliminate the heterogeneity between different modalities. Therefore, it is challenging to make the subsequent emotion class boundary learning. To tackle the above problems, we have proposed a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation, which can achieve information interaction between modalities and eliminate heterogeneity among modalities. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Specifically, we construct a graph structure for three modal features and perform contrastive representation learning on nodes with different emotions in the same modality and the same emotion in different modalities, which can improve the feature representation ability of nodes. Extensive experimental works show that the ARL-IIGCN method can significantly improve emotion recognition accuracy on IEMOCAP and MELD datasets.

Learning Robust Multi-Modal Representation for Multi-Label Emotion Recognition Via Adversarial Masking and Perturbation

Cross-Culture Multimodal Emotion Recognition With Adversarial Learning

Capturing Emotion Distribution for Multimedia Emotion Tagging

Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition

CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition

Label Distribution Adaptation for Multimodal Emotion Recognition with Multi-label Learning

Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching

A Versatile Multimodal Learning Framework For Zero-shot Emotion Recognition

Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios

Multimodal Emotion Recognition in Noisy Environment Based on Progressive Label Revision

Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognition

Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation

MALN: Multimodal Adversarial Learning Network for Conversational Emotion Recognition

Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo-labeling.

The Multi-modal Emotion Recognition Based on Text and Image