Abstract:<p>Social media has become indispensable to people's lives, where they can share their views and emotion with images and texts. Analyzing social images for sentiment prediction can help understand human social behavior and provide better recommendation results. Most current researches on image sentiment analysis have achieved quite good progress, which ignores the semantic correlation between an image and its corresponding descriptive sentences (caption). To capture the complementary multimodal information for joint sentiment classification, in this paper, we propose a novel cross-modal Semantic Content Correlation(SCC) method based on deep matching and hierarchical networks, which bridges the correlation between images and captions. Specifically, pre-trained convolutional neural networks (CNNs) are leveraged to encode the visual sub-regions contents, and a GloVe is employed to embed the textual semantic. Relying on visual contents and textual semantic, a joint attention network is proposed to learn the content correlation of the image and its caption, which is then exported as an image-text pair. To exploit the dependence of visual contents on textual semantic in caption effectively, the caption is processed by a Class-Aware Sentence Representation (CASR) network with a class dictionary, and a fully connected layer concatenates the outputs of CASR into a class-aware vector. Finally, the class-aware distributed vector is fed into an Inner-class Dependency Long Short-Term Memory network (IDLSTM) with the image–text pair as a query to further capture the cross-modal non-linear correlations for sentiment prediction. The performance of extensive experiments conducted on three datasets verifies the effectiveness of the model SCC.</p>

Multimodal sentiment analysis based on cross-instance graph neural networks

Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks.

DGFN Multimodal Emotion Analysis Model Based on Dynamic Graph Fusion Network

A Graph Convolution-Based Heterogeneous Fusion Network for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multi-Channel Attentive Graph Convolutional Network with Sentiment Fusion for Multimodal Sentiment Analysis

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Multimodal Hypergraph Learning For Microblog Sentiment Prediction

Cross-Modality Microblog Sentiment Prediction Via Bi-Layer Multimodal Hypergraph Learning

Multimodal sentiment analysis based on multiple attention

Cross-Modality Sentiment Analysis for Social Multimedia

Multimodal Sentiment Analysis Network Based on Distributional Transformation and Gated Cross-Modal Fusion

Image-text interaction graph neural network for image-text sentiment analysis

Various syncretic co‐attention network for multimodal sentiment analysis

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Temporal Graph Convolutional Network for Multimodal Sentiment Analysis

Multimodal Interaction and Fused Graph Convolution Network for Sentiment Classification of Online Reviews

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Capturing High-Level Semantic Correlations via Graph for Multimodal Sentiment Analysis

Multimodal Sentiment Analysis With Image-Text Interaction Network

Cross-modal image sentiment analysis via deep correlation of textual semantic