Abstract:Emotion Recognition in Conversations (ERC) aims to accurately identify the emotional labels of each utterance in a conversation, holding significant application value in human–computer interaction. Existing research suggests introducing commonsense knowledge (CSK) and multimodal information enhances model performance in ERC tasks. However, several challenges persist: (1) the neglect of complex psychological influences between utterances; (2) noise issues within modal information; (3) prediction challenges for emotion labels with few samples in different categories that exhibit semantic similarity but distinct emotional categories. To address the above problems, we propose a Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning (MKIN-MCL). Firstly, we establish a knowledge aggregation graph to capture the dependencies of commonsense knowledge (CSK) between utterances during a conversation. We actively aggregate relevant knowledge information to enhance text features. Simultaneously, we apply feature filters for acoustic and visual modalities to eliminate noise and enhance feature quality. Furthermore, we implement an interactive attention module by stacking designed Cross-modal Interactive Transformers (CITs) to continuously explore the relevance between the interacting parties in their respective semantic spaces, thus improving the effectiveness of modality interaction while reducing noise generated during the interaction. Lastly, we employ the Mixed Contrastive Learning (MCL) strategy to enhance the model’s ability to handle few-shot labels. This strategy utilizes unsupervised contrastive learning to improve the representation capability of the multimodal fusion features and supervised contrastive learning to extract information from few-shot labels. Extensive experiments on two benchmark datasets, IEMOCAP and MELD, validate the effectiveness and superiority of the proposed model.

An Enhanced Speech Emotion Recognition System Based On Discourse Information

Emotional Speech Clustering Based Robust Speaker Recognition System

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Cost-Sensitive Learning for Emotion Robust Speaker Recognition

Speech Emotion Recognition Based on Linear Discriminant Analysis and Support Vector Machine Decision Tree

Speech Emotion Recognition Based on Feature Selection and Extreme Learning Machine Decision Tree

Emotion Recognition From Noisy Speech

Speaker-Independent Speech Emotion Recognition Based On Cnn-Blstm And Multiple Svms

Construction and Evaluation of Mandarin Multimodal Emotional Speech Database

Research on Chinese Speech Emotion Recognition Based on Deep Neural Network and Acoustic Features

Affect-insensitive Speaker Recognition Systems Via Emotional Speech Clustering Using Prosodic Features

Multimodal Knowledge-enhanced Interactive Network with Mixed Contrastive Learning for Emotion Recognition in Conversation

Manifolds Based Emotion Recognition in Speech.

Emotion Recognition in Conversation Based on a Dynamic Complementary Graph Convolutional Network

EmoEars: an emotion recognition system for mandarin speech

Deep Learning and SVM-based Emotion Recognition from Chinese Speech for Smart Affective Services

Speech Emotion Recognition Using Deep Convolutional Neural Network and Discriminant Temporal Pyramid Matching

A Contextual Attention Network for Multimodal Emotion Recognition in Conversation

Speech Emotion Recognition Based on Multi-feature and Multi-lingual Fusion

Speech Emotion Recognition with Emotion-Pair Based Framework Considering Emotion Distribution Information in Dimensional Emotion Space.

Research on the Application of Speech Database based on Emotional Feature Extraction in International Chinese Education and Teaching