Abstract:In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals' attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model.

Multimodal Hypergraph Learning For Microblog Sentiment Prediction

Cross-Modality Microblog Sentiment Prediction Via Bi-Layer Multimodal Hypergraph Learning

Word Dictionary Emoticon Dictionary SentiBank : ANP Detector Library Microblog with labeled sentiment Testing microblogs Update Update W Update g

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Multimodal sentiment analysis based on cross-instance graph neural networks

Predicting Microblog Sentiments Via Weakly Supervised Multimodal Deep Learning.

Capturing High-Level Semantic Correlations via Graph for Multimodal Sentiment Analysis

Do Photos Help Express Our Feelings: Incorporating Multimodal Features into Microblog Sentiment Analysis.

Microblog Sentiment Classification with Heterogeneous Sentiment Knowledge

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

A Multi-dimension and Multi-granularity Feature Fusion Method for Chinese Microblog Sentiment Classification

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Multimodal Sentiment Recognition With Multi-Task Learning

Multimodal Sentiment Analysis of Graphic Texts Based on Multicategorical Relative Fusion

Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis

A Joint Model for Chinese Microblog Sentiment Analysis.

Multi-view sentiment classification of microblogs based on semantic features

Multichannel Multimodal Emotion Analysis of Cross-Modal Feedback Interactions Based on Knowledge Graph

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations