Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

Modeling both Intra- and Inter-modal Influence for Real-Time Emotion Detection in Conversations

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

MFDR: Multiple-stage Fusion and Dynamically Refined Network for Multimodal Emotion Recognition

Investigating Multisensory Integration in Emotion Recognition Through Bio-Inspired Computational Models

EffMulti: Efficiently Modeling Complex Multimodal Interactions for Emotion Analysis

Bridging the Emotional Semantic Gap via Multimodal Relevance Estimation

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

MMDAG: Multimodal Directed Acyclic Graph Network for Emotion Recognition in Conversation

InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis

A multimodal shared network with a cross-modal distribution constraint for continuous emotion recognition

Dynamic Causal Disentanglement Model for Dialogue Emotion Detection

Multimodal interaction enhanced representation learning for video emotion recognition

Contextual and Cross-Modal Interaction for Multi-Modal Speech Emotion Recognition

Dissecting emotion and user influence in social media communities: An interaction modeling approach

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Bridging Discrete and Continuous: A Multimodal Strategy for Complex Emotion Detection

Inferring User Emotive State Changes in Realistic Human-Computer Conversational Dialogs.

AIMDiT: Modality Augmentation and Interaction via Multimodal Dimension Transformation for Emotion Recognition in Conversations

Learning Interactions among Multi-Channel Sequences with Dynamical Influence Models

Modeling Emotion Influence Using Attention-based Graph Convolutional Recurrent Network