Abstract:The recent booming of artificial intelligence (AI) applications, e.g., affective robots, human-machine interfaces, autonomous vehicles, etc., has produced a great number of multi-modal records of human communication. Such data often carry latent subjective users’ attitudes and opinions, which provides a practical and feasible path to realize the connection between human emotion and intelligence services. Sentiment and emotion analysis of multi-modal records is of great value to improve the intelligent level of affective services. However, how to find an optimal manner to learn people’s sentiment and emotion representations has been a difficult problem, since both of them involve subtle mind activity. To solve this problem, a lot of approaches have been published, but most of which are insufficient to mine sentiment and emotion, since they have treated sentiment analysis and emotion recognition as two separate tasks. The interaction between them has been neglected, which limits the efficiency of sentiment and emotion representation learning. In this work, emotion is seen as the external expression of sentiment, while sentiment is the essential nature of emotion. We thus argue that they are strongly related to each other where one’s judgment helps the decision of the other. The key challenges are multi-modal fused representation and the interaction between sentiment and emotion. To solve such issues, we design an external knowledge enhanced multi-task representation learning network, termed KAMT. The major elements contain two attention mechanisms, which are inter-modal and inter-task attentions and an external knowledge augmentation layer. The external knowledge augmentation layer is used to extract the vector of the participant’s gender, age, occupation and that of overall color or shape. The main use of inter-modal attention is to capture effective multi-modal fused features. Inter-task attention is designed to model the correlation between sentiment analysis and emotion classification. We perform experiments on three widely used datasets, and the experimental performance proves the effectiveness of the KAMT model.

Emotion and sentiment analysis for intelligent customer service conversation using a multi-task ensemble framework

All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework

Sentiments analysis for intelligent customer service dialogue using hybrid word embedding and stacking ensemble

Multi-modal Sentiment and Emotion Joint Analysis with a Deep Attentive Multi-task Learning Model

Visual-Audio Emotion Recognition Based on Multi-Task and Ensemble Learning with Multiple Features

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

Multi-Task Multi-Head Attention Memory Network for Fine-Grained Sentiment Analysis.

Sentiment Classification in Customer Service Dialogue with Topic-Aware Multi-Task Learning

Text Emotion Distribution Learning Via Multi-Task Convolutional Neural Network

A Deep Multi-task Contextual Attention Framework for Multi-modal Affect Analysis

SENTIMENT CLASSIFICATION THROUGH COLLABORATIVE MULTI-INPUT AND MULTI-TASK LEARNING

Aspect-Level Sentiment Analysis of Customer Reviews Based on Neural Multi-task Learning

Multiple Scene Sentiment Analysis Based on Chinese Speech and Text

Chinese Multicategory Sentiment of E-Commerce Analysis Based on Deep Learning

Multi-Task Learning Model Based on Multi-Scale CNN and LSTM for Sentiment Classification

Multi-feature fusion and dual-channel networks for sentiment analysis

A Multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversations

A Multi-model Fusion Framework Based on Deep Learning for Sentiment Classification.