Abstract:Sentiment analysis is crucial for many social media analytic tasks. Earlier researches mainly focus on single modality, e.g., text description or visual content. Recently, more and more works pay attention to the incorporation of multiple modalities. Different from the traditional image database, social images usually interconnect with each other, which makes the sentiment analysis nontrivial. Most existing methods consider different images independently, which cannot be directly applied to the interconnected images. In this paper, we propose a novel Hierarchical Deep Fusion (HDF) model to explore the cross-modal correlations among images, texts, and their social links, which can learn comprehensive and complementary features for more effective sentiment analysis. Specifically, we combine the visual content with different semantic fragments of textual content through a three-level hierarchical LSTMs (H-LSTMs) to learn the inter-modal correlations between image and text at different levels. To exploit the link information effectively, the linkages among social images are modeled by a weighted relation network and each node is embedded into a distributed vector. Then, the extracted image–text features and node embeddings are fused by a Multi-Layer Perceptron (MLP) to further capture the non-linear cross-modal correlations for sentiment prediction. Comprehensive experiments are conducted to demonstrate the effectiveness of our approach on both machine weakly labeled and manually labeled datasets.

Image Sentiment Analysis Method Based on Multi-Level Feature Fusion

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Multi-level fusion with deep neural networks for multimodal sentiment classification

Sentiment Analysis of Social Images Via Hierarchical Deep Fusion of Content and Links.

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

A Multi-level Style Feature Model for Visual Sentiment Analysis

A Multi-model Fusion Framework Based on Deep Learning for Sentiment Classification.

Multi-Feature Fusion for Multimodal Attentive Sentiment Analysis.

Deep Learning Based Weighted Feature Fusion Approach for Sentiment Analysis

Image-text sentiment analysis via deep multimodal attentive fusion.

Attention-based multi-level image and text sentiment analysis

Exploring Multimodal Multiscale Features for Sentiment Analysis Using Fuzzy-Deep Neural Network Learning

A Multi-feature Fusion and SSAE-Based Deep Network for Image Semantic Recognition

Sentiment Analysis Methods Based on Multi-Modal Fusion and Deep Learning

Interactive Fusion Network with Recurrent Attention for Multimodal Aspect-based Sentiment Analysis.

MFSC: A Multimodal Aspect-Level Sentiment Classification Framework with Multi-Image Gate and Fusion Networks

Recognizing Image Semantic Information Through Multi-Feature Fusion and SSAE-Based Deep Network

VISUAL AND TEXTUAL SENTIMENT ANALYSIS USING DEEP FUSION CONVOLUTIONAL NEURAL NETWORKS

Learning Multi-level Representations for Image Emotion Recognition in the Deep Convolutional Network

Graphic image classification method based on an attention mechanism and fusion of multilevel and multiscale deep features

Deep Multi-Level Fusion Network for Multi-Source Image Pixel-Wise Classification