Abstract:Multimodal aspect-based sentiment analysis (MABSA) aims to determine the sentiment polarity of each aspect mentioned in the text based on multimodal content. Various approaches have been proposed to model multimodal sentiment features for each aspect via modal interactions. However, most existing approaches have two shortcomings: (1) The representation gap between textual and visual modalities may increase the risk of misalignment in modal interactions; (2) In some examples where the image is not related to the text, the visual information may not enrich the textual modality when learning aspect-based sentiment features. In such cases, blindly leveraging information from visual modal may introduce noises in reasoning the aspect-based sentiment expressions. To tackle the shortcomings mentioned above, we propose an end-to-end MABSA framework with image conversion and noise filtration. Specifically, to bridge the representation gap in different modalities, we attempt to translate images into the input space of a pre-trained language model (PLM). To this end, we develop an image-to-text conversion module that can convert an image to an implicit sequence of token embedding. Moreover, an aspect-oriented filtration module is devised to alleviate the noise in the implicit token embeddings, which consists of two attention operations. The former aims to create an enhanced aspect embedding as a query, and the latter seeks to use this query to retrieve relevant auxiliary information from the implicit token embeddings to supplement the textual content. After filtering the noise, we leverage a PLM to encode the text, aspect, and image prompt derived from filtered implicit token embeddings as sentiment features to perform aspect-based sentiment prediction. Experimental results on two MABSA datasets show that our framework achieves state-of-the-art performance. Furthermore, extensive experimental analysis demonstrates the proposed framework has superior robustness and efficiency.

Image Sentiment Prediction Based on Textual Descriptions with Adjective Noun Pairs

Visual sentiment analysis based on image caption and adjective–noun–pair description

Large-scale Visual Sentiment Ontology and Detectors Using Adjective Noun Pairs

Beyond Object Recognition: Visual Sentiment Analysis with Deep Coupled Adjective and Noun Neural Networks

SentiBank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content.

More cat than cute? Interpretable Prediction of Adjective-Noun Pairs

DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks

SentiCap: Generating Image Descriptions with Sentiments

Sentiment Prediction in Scene Images Via Convolutional Neural Networks

Transfer Correlation Between Textual Content to Images for Sentiment Analysis

Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Sentiment Classification Using Negative and Intensive Sentiment Supplement Information

Image-to-Text Conversion and Aspect-Oriented Filtration for Multimodal Aspect-Based Sentiment Analysis

Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data

Object-Based Visual Sentiment Concept Analysis and Application

Sentiment Analysis of Social Images Via Hierarchical Deep Fusion of Content and Links.

Word Dictionary Emoticon Dictionary SentiBank : ANP Detector Library Microblog with labeled sentiment Testing microblogs Update Update W Update g

Image Captioning with Inherent Sentiment

Image Captioning by Incorporating Affective Concepts Learned from Both Visual and Textual Components.

Deep Coordinated Textual and Visual Network for Sentiment-Oriented Cross-Modal Retrieval

The Role of Visual Attention in Sentiment Prediction