Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

Dual Learning for Conversational Emotion Recognition and Emotional Response Generation

CDL: Curriculum Dual Learning for Emotion-Controllable Response Generation

Emotion recognition in conversations with emotion shift detection based on multi-task learning

Real-Time Video Emotion Recognition Based on Reinforcement Learning and Domain Knowledge

From Chatter to Matter: Addressing Critical Steps of Emotion Recognition Learning in Task-oriented Dialogue

InstructERC: Reforming Emotion Recognition in Conversation with Multi-task Retrieval-Augmented Large Language Models

Hybrid Multi-Task Learning for End-To-End Multimodal Emotion Recognition

A Dual-Stream Recurrence-Attention Network With Global-Local Awareness for Emotion Recognition in Textual Dialog

Model-Level Dual Learning.

Constructing Emotion Consensus and Utilizing Unpaired Data for Empathetic Dialogue Generation

Speaker-Guided Encoder-Decoder Framework for Emotion Recognition in Conversation

EmotionIC: emotional inertia and contagion-driven dependency modeling for emotion recognition in conversation

Human-Robot Emotional Interaction Model Based on Reinforcement Learning

Conversational transfer learning for emotion recognition

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

CKERC : Joint Large Language Models with Commonsense Knowledge for Emotion Recognition in Conversation

Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

SI-LSTM: Speaker Hybrid Long-short Term Memory and Cross Modal Attention for Emotion Recognition in Conversation

Curriculum Learning Meets Directed Acyclic Graph for Multimodal Emotion Recognition

Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognition