Abstract:These days, social media users tend to express their feelings through sharing images online. Capturing the emotions embedded in these social images involves great research challenges and practical values. Most existing works concentrate on extracting the visual feature from a global view, while ignoring the fact that visual objects are also rich in emotion. How to leverage the multilevel visual features to improve the sentiment analysis performance is important yet challenging. Besides, existing works view each social image as an independent sample while ignoring the rich correlations among social images, which may be helpful in detecting visual emotion. In this article, we propose a novel model called social relations-guided multiattention networks (SRGMANs) to incorporate both the multilevel (region-level and object-level) visual features of a single image and the correlations among multiple social images to conduct visual sentiment analysis. Specifically, we first construct a heterogeneous network consisting of various types of social relations and introduce a heterogeneous network embedding method to learn the network representation for each image. Then, two visual attention branches (region attention network and object attention network) are devised to extract emotional and discriminative visual features. For each branch, we design a self-attention module to capture the emotional dependencies among visual parts. Besides, a network-guided attention module is also designed in each branch to focus on more network-related emotional visual parts with the guidance of the topology information. Finally, the attended visual features from the two attention models, together with network representation features, are combined within a holistic framework to predict the sentiment of social images. Extensive experiments demonstrate the superiority of our model on three benchmark datasets.

Visual Sentiment Analysis with Network in Network

Visualizing and Understanding Neural Models in NLP

Visual And Textual Sentiment Analysis Of A Microblog Using Deep Convolutional Neural Networks

From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction

Beyond Object Recognition: Visual Sentiment Analysis with Deep Coupled Adjective and Noun Neural Networks

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networks

VISUAL AND TEXTUAL SENTIMENT ANALYSIS USING DEEP FUSION CONVOLUTIONAL NEURAL NETWORKS

Sentiment Prediction in Scene Images Via Convolutional Neural Networks

Visual Sentiment Analysis Using Deep Learning Models with Social Media Data

Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction

SentiNet: Mining Visual Sentiment from Scratch.

Visual sentiment analysis with semantic correlation enhancement

Visual sentiment analysis based on image caption and adjective–noun–pair description

Weakly Supervised Coupled Networks for Visual Sentiment Analysis

Visual Sentiment Analysis With Social Relations-Guided Multiattention Networks

The Role of Visual Attention in Sentiment Prediction

Visual sentiment analysis using data-augmented deep transfer learning techniques

A deep learning architecture of RA-DLNet for visual sentiment analysis

Visual-textual Sentiment Classification with Bi-Directional Multi-Level Attention Networks

Unsupervised Feature Learning Assisted Visual Sentiment Analysis