Abstract:These days, social media users tend to express their feelings through sharing images online. Capturing the emotions embedded in these social images involves great research challenges and practical values. Most existing works concentrate on extracting the visual feature from a global view, while ignoring the fact that visual objects are also rich in emotion. How to leverage the multilevel visual features to improve the sentiment analysis performance is important yet challenging. Besides, existing works view each social image as an independent sample while ignoring the rich correlations among social images, which may be helpful in detecting visual emotion. In this article, we propose a novel model called social relations-guided multiattention networks (SRGMANs) to incorporate both the multilevel (region-level and object-level) visual features of a single image and the correlations among multiple social images to conduct visual sentiment analysis. Specifically, we first construct a heterogeneous network consisting of various types of social relations and introduce a heterogeneous network embedding method to learn the network representation for each image. Then, two visual attention branches (region attention network and object attention network) are devised to extract emotional and discriminative visual features. For each branch, we design a self-attention module to capture the emotional dependencies among visual parts. Besides, a network-guided attention module is also designed in each branch to focus on more network-related emotional visual parts with the guidance of the topology information. Finally, the attended visual features from the two attention models, together with network representation features, are combined within a holistic framework to predict the sentiment of social images. Extensive experiments demonstrate the superiority of our model on three benchmark datasets.

The Role of Visual Attention in Sentiment Prediction

Emotional Attention: A Study of Image Sentiment and Visual Attention

Visual Sentiment Analysis With Social Relations-Guided Multiattention Networks

Attention-based multi-level image and text sentiment analysis

Emotional attention: From eye tracking to computational modeling

From Pixels to Sentiment: Fine-tuning CNNs for Visual Sentiment Prediction

Visual sentiment analysis based on image caption and adjective–noun–pair description

Visual sentiment analysis with semantic correlation enhancement

Senti-Attend: Image Captioning using Sentiment and Attention

VisdaNet: Visual Distillation and Attention Network for Multimodal Sentiment Classification

A Multi-Attentive Pyramidal Model for Visual Sentiment Analysis

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Context-aware Attention Network for Predicting Image Aesthetic Subjectivity

Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction

Visual Emotion Analysis Via Affective Semantic Concept Discovery

Sentiment Prediction in Scene Images Via Convolutional Neural Networks

A Lexicon-Based Supervised Attention Model for Neural Sentiment Analysis.

Prediction of emotion distribution of images based on weighted K-nearest neighbor-attention mechanism

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention