Abstract:These days, social media users tend to express their feelings through sharing images online. Capturing the emotions embedded in these social images involves great research challenges and practical values. Most existing works concentrate on extracting the visual feature from a global view, while ignoring the fact that visual objects are also rich in emotion. How to leverage the multilevel visual features to improve the sentiment analysis performance is important yet challenging. Besides, existing works view each social image as an independent sample while ignoring the rich correlations among social images, which may be helpful in detecting visual emotion. In this article, we propose a novel model called social relations-guided multiattention networks (SRGMANs) to incorporate both the multilevel (region-level and object-level) visual features of a single image and the correlations among multiple social images to conduct visual sentiment analysis. Specifically, we first construct a heterogeneous network consisting of various types of social relations and introduce a heterogeneous network embedding method to learn the network representation for each image. Then, two visual attention branches (region attention network and object attention network) are devised to extract emotional and discriminative visual features. For each branch, we design a self-attention module to capture the emotional dependencies among visual parts. Besides, a network-guided attention module is also designed in each branch to focus on more network-related emotional visual parts with the guidance of the topology information. Finally, the attended visual features from the two attention models, together with network representation features, are combined within a holistic framework to predict the sentiment of social images. Extensive experiments demonstrate the superiority of our model on three benchmark datasets.

User-guided Hierarchical Attention Network for Multi-modal Social Image Popularity Prediction.

A Hierarchical Attention Model for Social Contextual Image Recommendation

Explainable Social Contextual Image Recommendation with Hierarchical Attention.

Analyzing multimodal public sentiment based on hierarchical semantic attentional network

Semantic Guided Level-Category Hybrid Prediction Network for Hierarchical Image Classification.

Hierarchical Attention Neural Network for Information Cascade Prediction

Hierarchical Gumbel Attention Network for Text-based Person Search

Hierarchical Attention Networks for Multimodal Machine Learning

UniAR: A Unified model for predicting human Attention and Responses on visual content

User-Video Co-Attention Network for Personalized Micro-video Recommendation

Hierarchical Attention Networks for Image Classification of Remote Sensing Images Based on Visual Q&A Methods

Personalized Fashion Recommendation with Visual Explanations Based on Multimodal Attention Network

Visual-Textual Sentiment Analysis Enhanced by Hierarchical Cross-Modality Interaction

Multimodal Unified Attention Networks for Vision-and-Language Interactions

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

Multimodal Video Saliency Analysis with User-Biased Information.

MHSAN: Multi-view hierarchical self-attention network for 3D shape recognition

HierCas: Hierarchical Temporal Graph Attention Networks for Popularity Prediction in Information Cascades

Multimodal-enhanced hierarchical attention network for video captioning

How to Become Instagram Famous: Post Popularity Prediction with Dual-Attention

Visual Sentiment Analysis With Social Relations-Guided Multiattention Networks