Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

SMIN: Semi-supervised Multi-modal Interaction Network for Conversational Emotion Recognition

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

Semi-IIN: Semi-supervised Intra-inter modal Interaction Learning Network for Multimodal Sentiment Analysis

SIA-Net: Sparse Interactive Attention Network for Multimodal Emotion Recognition

Smile: Spiking Multi-Modal Interactive Label-Guided Enhancement Network for Emotion Recognition

Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition

Semantic Enhancement Network Integrating Label Knowledge for Multi-modal Emotion Recognition

Multi-Grained Multimodal Interaction Network for Sentiment Analysis.

Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching

Dynamic Interactive Multiview Memory Network for Emotion Recognition in Conversation

A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation.

Multimodal Speech Emotion Recognition Based on Multi-Scale MFCCs and Multi-View Attention Mechanism

MMGCN: Multimodal Fusion Via Deep Graph Convolution Network for Emotion Recognition in Conversation

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

Multimodal Emotion Classification with Multi-Level Semantic Reasoning Network

Multimodal interaction enhanced representation learning for video emotion recognition

A Multi-Level Alignment and Cross-Modal Unified Semantic Graph Refinement Network for Conversational Emotion Recognition

A Deep Spatiotemporal Interaction Network for Multimodal Sentimental Analysis and Emotion Recognition

A Dual Attention-based Modality-Collaborative Fusion Network for Emotion Recognition

MFGCN: Multimodal fusion graph convolutional network for speech emotion recognition