Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

PMTL: Personality-Aware Multitask Learning Model for Emotion Recognition in Conversation

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Multitask Learning for Emotion and Personality Detection

Multi-Modal Attentive Prompt Learning for Few-shot Emotion Recognition in Conversations

A Persona-Infused Cross-Task Graph Network for Multimodal Emotion Recognition with Emotion Shift Detection in Conversations

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Multi-modal emotion recognition using tensor decomposition fusion and self-supervised multi-tasking

Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation

Emotion recognition in conversations with emotion shift detection based on multi-task learning

Transfer Learning for Personality Perception via Speech Emotion Recognition

A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

MMATERIC: Multi-Task Learning and Multi-Fusion for AudioText Emotion Recognition in Conversation

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

PIRNet: Personality-Enhanced Iterative Refinement Network for Emotion Recognition in Conversation

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

MALN: Multimodal Adversarial Learning Network for Conversational Emotion Recognition