Abstract:In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals' attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model.

Unsupervised multimodal learning for image-text relation classification in tweets

Modality-invariant Temporal Representation Learning for Multimodal Sentiment Classification

Semi-Supervised Dual Relation Learning for Multi-Label Classification

Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach

Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks

Social Image-text Sentiment Classification With Cross-Modal Consistency and Knowledge Distillation

RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER.

A multi‐label social short text classification method based on contrastive learning and improved ml‐KNN

I2SRM: Intra- and Inter-Sample Relationship Modeling for Multimodal Information Extraction

Graph-based Multimodal Semi-Supervised Image Classification

Joint Intermodal and Intramodal Label Transfers for Extremely Rare or Unseen Classes

Contrastive Graph Multimodal Model for Text Classification in Videos

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Multimodal Sentiment Analysis With Image-Text Interaction Network

Image-text Retrieval: A Survey on Recent Research and Development

TCGM: an Information-Theoretic Framework for Semi-Supervised Multi-Modality Learning

Transfer Correlation Between Textual Content to Images for Sentiment Analysis

A multimodal sentiment recognition method based on attention mechanism

Semi-supervised Relational Topic Model for Weakly Annotated Image Recognition in Social Media

A Semi-supervised Method for Classification of Semantic Relation Between Nominals

Social-Correlation Based Mutual Reinforcement for Short Text Classification and User Interest Tagging