Abstract:The recent booming of artificial intelligence (AI) applications, e.g., affective robots, human-machine interfaces, autonomous vehicles, etc., has produced a great number of multi-modal records of human communication. Such data often carry latent subjective users’ attitudes and opinions, which provides a practical and feasible path to realize the connection between human emotion and intelligence services. Sentiment and emotion analysis of multi-modal records is of great value to improve the intelligent level of affective services. However, how to find an optimal manner to learn people’s sentiment and emotion representations has been a difficult problem, since both of them involve subtle mind activity. To solve this problem, a lot of approaches have been published, but most of which are insufficient to mine sentiment and emotion, since they have treated sentiment analysis and emotion recognition as two separate tasks. The interaction between them has been neglected, which limits the efficiency of sentiment and emotion representation learning. In this work, emotion is seen as the external expression of sentiment, while sentiment is the essential nature of emotion. We thus argue that they are strongly related to each other where one’s judgment helps the decision of the other. The key challenges are multi-modal fused representation and the interaction between sentiment and emotion. To solve such issues, we design an external knowledge enhanced multi-task representation learning network, termed KAMT. The major elements contain two attention mechanisms, which are inter-modal and inter-task attentions and an external knowledge augmentation layer. The external knowledge augmentation layer is used to extract the vector of the participant’s gender, age, occupation and that of overall color or shape. The main use of inter-modal attention is to capture effective multi-modal fused features. Inter-task attention is designed to model the correlation between sentiment analysis and emotion classification. We perform experiments on three widely used datasets, and the experimental performance proves the effectiveness of the KAMT model.

A Multitask Multimodal Ensemble Model for Sentiment- and Emotion-Aided Tweet Act Classification

An Emotion based Sentiment Analysis on Twitter Dataset

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

A soft voting ensemble learning-based approach for multimodal sentiment analysis

Multi-class Emotion AI by reconstructing linguistic context of words

Target-Dependent Multimodal Sentiment Analysis Via Employing Visual-to Emotional-Caption Translation Network using Visual-Caption Pairs

Emotion Detection and Analysis on Social Media

Affective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification

Multimodal, multiview and multitasking depression detection framework endorsed with auxiliary sentiment polarity and emotion detection

Multimodal Sentiment Analysis To Explore the Structure of Emotions

All-in-One: Emotion, Sentiment and Intensity Prediction Using a Multi-Task Ensemble Framework

Leveraging distant supervision and deep learning for twitter sentiment and emotion classification

Task-agnostic representation learning of multimodal twitter data for downstream applications

Emoticon Smoothed Language Models for Twitter Sentiment Analysis

A Multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversations

An efficient multimodal sentiment analysis in social media using hybrid optimal multi-scale residual attention network

Fine-Tuning BERT Based Approach for Multi-Class Sentiment Analysis on Twitter Emotion Data

Image-Text Multimodal Emotion Classification via Multi-View Attentional Network

Sentiment Analysis and Sarcasm Detection using Deep Multi-Task Learning

Advancing Twitter Sentiment Analysis: An Ensemble Approach with Transformer-XL, RoBERTa, and XGBoost