Abstract:The recent booming of artificial intelligence (AI) applications, e.g., affective robots, human-machine interfaces, autonomous vehicles, etc., has produced a great number of multi-modal records of human communication. Such data often carry latent subjective users’ attitudes and opinions, which provides a practical and feasible path to realize the connection between human emotion and intelligence services. Sentiment and emotion analysis of multi-modal records is of great value to improve the intelligent level of affective services. However, how to find an optimal manner to learn people’s sentiment and emotion representations has been a difficult problem, since both of them involve subtle mind activity. To solve this problem, a lot of approaches have been published, but most of which are insufficient to mine sentiment and emotion, since they have treated sentiment analysis and emotion recognition as two separate tasks. The interaction between them has been neglected, which limits the efficiency of sentiment and emotion representation learning. In this work, emotion is seen as the external expression of sentiment, while sentiment is the essential nature of emotion. We thus argue that they are strongly related to each other where one’s judgment helps the decision of the other. The key challenges are multi-modal fused representation and the interaction between sentiment and emotion. To solve such issues, we design an external knowledge enhanced multi-task representation learning network, termed KAMT. The major elements contain two attention mechanisms, which are inter-modal and inter-task attentions and an external knowledge augmentation layer. The external knowledge augmentation layer is used to extract the vector of the participant’s gender, age, occupation and that of overall color or shape. The main use of inter-modal attention is to capture effective multi-modal fused features. Inter-task attention is designed to model the correlation between sentiment analysis and emotion classification. We perform experiments on three widely used datasets, and the experimental performance proves the effectiveness of the KAMT model.

Interactive Multi-Grained Joint Model for Targeted Sentiment Analysis

Prior-Bert and Multi-Task Learning for Target-Aspect-Sentiment Joint Detection

Joint Learning for Targeted Sentiment Analysis.

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysis

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Multiple-element joint detection for Aspect-Based Sentiment Analysis

Multi-Task Multi-Head Attention Memory Network for Fine-Grained Sentiment Analysis.

Jointly Discovering Fine-grained and Coarse-grained Sentiments Via Topic Modeling.

Jointly Modeling Aspect and Sentiment with Dynamic Heterogeneous Graph Neural Networks

Learning Explicit and Implicit Structures for Targeted Sentiment Analysis

Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction

Self-Adaptive Representation Learning Model for Multi-Modal Sentiment and Sarcasm Joint Analysis

Target-oriented Multimodal Sentiment Classification by Using Topic Model and Gating Mechanism.

Sentiment interpretability analysis on Chinese texts employing multi-task and knowledge base

Jointly Learning Sentiment, Keyword and Opinion Leader in Social Reviews

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Effective LSTMs for Target-Dependent Sentiment Classification

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification