Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion Recognition

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

ERNetCL: A novel emotion recognition network in textual conversation based on curriculum learning strategy

Conversational emotion recognition studies based on graph convolutional neural networks and a dependent syntactic analysis

Multi-Scale Receptive Field Graph Model for Emotion Recognition in Conversations

Multimodal Emotional Classification Based on Meaningful Learning

Multiple Knowledge-Enhanced Interactive Graph Network for Multimodal Conversational Emotion Recognition

A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning

Cluster-Level Contrastive Learning for Emotion Recognition in Conversations

Dynamic Graph Neural Ordinary Differential Equation Network for Multi-modal Emotion Recognition in Conversation

Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversation

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognition

Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations

Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction

Deep Graph-Recurrent Model for Emotion Recognition in Conversation Using Fully Connected Directed Acyclic

Adaptive Graph Learning for Multimodal Conversational Emotion Detection

Multimodal Fusion via Hypergraph Autoencoder and Contrastive Learning for Emotion Recognition in Conversation

DER-GCN: Dialog and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialog Emotion Recognition

Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge