Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

Knowing What and Why: Causal Emotion Entailment for Emotion Recognition in Conversation

Dialogue emotion model based on local–global context encoder and commonsense knowledge fusion attention

UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause

Neutral Utterances are Also Causes: Enhancing Conversational Causal Emotion Entailment with Social Commonsense Knowledge

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Context De-confounded Emotion Recognition

ECR-Chain: Advancing Generative Language Models to Better Emotion-Cause Reasoners through Reasoning Chains

Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

A machine reading comprehension framework for recognizing emotion cause in conversations

PAGE: A Position-Aware Graph-Based Model for Emotion Cause Entailment in Conversation

ERNetCL: A novel emotion recognition network in textual conversation based on curriculum learning strategy

Contextual Information and Commonsense Based Prompt for Emotion Recognition in Conversation

Recognizing Conditional Causal Relationships about Emotions and Their Corresponding Conditions

Enhancing Emotion Recognition in Conversation Via Multi-view Feature Alignment and Memorization

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation

A Benchmark for Hierarchical Emotion Cause Extraction in Spoken Dialogues

Emotion recognition in conversations with emotion shift detection based on multi-task learning

Robust Emotion Recognition in Context Debiasing

Emotion Recognition in Conversation Based on a Dynamic Complementary Graph Convolutional Network