Abstract:With the release of increasing open-source emotion recognition datasets on social media platforms and the rapid development of computing resources, multimodal emotion recognition tasks (MER) have begun to receive widespread research attention. The MER task extracts and fuses complementary semantic information from different modalities, which can classify the speaker's emotions. However, the existing feature fusion methods have usually mapped the features of different modalities into the same feature space for information fusion, which can not eliminate the heterogeneity between different modalities. Therefore, it is challenging to make the subsequent emotion class boundary learning. To tackle the above problems, we have proposed a novel Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive for Multimodal Emotion Recognition (AR-IIGCN) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features through adversarial representation, which can achieve information interaction between modalities and eliminate heterogeneity among modalities. Thirdly, we introduce contrastive graph representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Specifically, we construct a graph structure for three modal features and perform contrastive representation learning on nodes with different emotions in the same modality and the same emotion in different modalities, which can improve the feature representation ability of nodes. Extensive experimental works show that the ARL-IIGCN method can significantly improve emotion recognition accuracy on IEMOCAP and MELD datasets.

Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples

Improving Multimodal Emotion Recognition by Leveraging Acoustic Adaptation and Visual Alignment

Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning.

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learning

Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognition

Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model

First-order Multi-label Learning with Cross-modal Interactions for Multimodal Emotion Recognition

A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Audio-Guided Fusion Techniques for Multimodal Emotion Analysis

Contrastive Regularization for Multimodal Emotion Recognition Using Audio and Text

MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition

Multi-modal Continuous Dimensional Emotion Recognition Using Recurrent Neural Network and Self-Attention Mechanism

Memobert: Pre-Training Model with Prompt-Based Learning for Multimodal Emotion Recognition

A Versatile Multimodal Learning Framework For Zero-shot Emotion Recognition

Emotion recognition based on multi-modal electrophysiology multi-head attention Contrastive Learning

Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning