Abstract:The recent booming of artificial intelligence (AI) applications, e.g., affective robots, human-machine interfaces, autonomous vehicles, etc., has produced a great number of multi-modal records of human communication. Such data often carry latent subjective users’ attitudes and opinions, which provides a practical and feasible path to realize the connection between human emotion and intelligence services. Sentiment and emotion analysis of multi-modal records is of great value to improve the intelligent level of affective services. However, how to find an optimal manner to learn people’s sentiment and emotion representations has been a difficult problem, since both of them involve subtle mind activity. To solve this problem, a lot of approaches have been published, but most of which are insufficient to mine sentiment and emotion, since they have treated sentiment analysis and emotion recognition as two separate tasks. The interaction between them has been neglected, which limits the efficiency of sentiment and emotion representation learning. In this work, emotion is seen as the external expression of sentiment, while sentiment is the essential nature of emotion. We thus argue that they are strongly related to each other where one’s judgment helps the decision of the other. The key challenges are multi-modal fused representation and the interaction between sentiment and emotion. To solve such issues, we design an external knowledge enhanced multi-task representation learning network, termed KAMT. The major elements contain two attention mechanisms, which are inter-modal and inter-task attentions and an external knowledge augmentation layer. The external knowledge augmentation layer is used to extract the vector of the participant’s gender, age, occupation and that of overall color or shape. The main use of inter-modal attention is to capture effective multi-modal fused features. Inter-task attention is designed to model the correlation between sentiment analysis and emotion classification. We perform experiments on three widely used datasets, and the experimental performance proves the effectiveness of the KAMT model.

A Supplementary Feature Set for Sentiment Analysis in Japanese Dialogues

Coupling Sentiment and Arousal Analysis Towards an Affective Dialogue Manager

A Mood Semantic Awareness Model for Emotional Interactive Robots

Building a Dialogue Corpus Annotated with Expressed and Experienced Emotions

A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild

Affective Common Sense Knowledge Acquisition for Sentiment Analysis.

USA: Universal Sentiment Analysis Model & Construction of Japanese Sentiment Text Classification and Part of Speech Dataset

Minimally Supervised Learning of Affective Events Using Discourse Relations

Affective Behaviour Analysis via Integrating Multi-Modal Knowledge

AdMISC: Advanced Multi-Task Learning and Feature-Fusion for Emotional Support Conversation

Deep Sentiment Features of Context and Faces for Affective Video Analysis

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification

A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognition

Infusing Emotions into Task-oriented Dialogue Systems: Understanding, Management, and Generation

Affect Detection from Semantic Interpretation of Drama Improvisation.

Acknowledgment of Emotional States: Generating Validating Responses for Empathetic Dialogue

Toward a Dialogue System Using a Large Language Model to Recognize User Emotions with a Camera

Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations

PSentScore: Evaluating Sentiment Polarity in Dialogue Summarization

Affect Recognition in Conversations Using Large Language Models