Abstract:Multimodal aspect-based sentiment analysis (MABSA) aims to determine the sentiment polarity of each aspect mentioned in the text based on multimodal content. Various approaches have been proposed to model multimodal sentiment features for each aspect via modal interactions. However, most existing approaches have two shortcomings: (1) The representation gap between textual and visual modalities may increase the risk of misalignment in modal interactions; (2) In some examples where the image is not related to the text, the visual information may not enrich the textual modality when learning aspect-based sentiment features. In such cases, blindly leveraging information from visual modal may introduce noises in reasoning the aspect-based sentiment expressions. To tackle the shortcomings mentioned above, we propose an end-to-end MABSA framework with image conversion and noise filtration. Specifically, to bridge the representation gap in different modalities, we attempt to translate images into the input space of a pre-trained language model (PLM). To this end, we develop an image-to-text conversion module that can convert an image to an implicit sequence of token embedding. Moreover, an aspect-oriented filtration module is devised to alleviate the noise in the implicit token embeddings, which consists of two attention operations. The former aims to create an enhanced aspect embedding as a query, and the latter seeks to use this query to retrieve relevant auxiliary information from the implicit token embeddings to supplement the textual content. After filtering the noise, we leverage a PLM to encode the text, aspect, and image prompt derived from filtered implicit token embeddings as sentiment features to perform aspect-based sentiment prediction. Experimental results on two MABSA datasets show that our framework achieves state-of-the-art performance. Furthermore, extensive experimental analysis demonstrates the proposed framework has superior robustness and efficiency.

FMCF: Few-shot Multimodal Aspect-Based Sentiment Analysis Framework Based on Contrastive Finetuning

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

Image-to-Text Conversion and Aspect-Oriented Filtration for Multimodal Aspect-Based Sentiment Analysis

MSFNet: modality smoothing fusion network for multimodal aspect-based sentiment analysis

Hierarchical Fusion Network with Enhanced Knowledge and Contrastive Learning for Multimodal Aspect-Based Sentiment Analysis on Social Media

A Simple Yet Effective Framework for Few-Shot Aspect-Based Sentiment Analysis.

Few-shot Aspect Category Sentiment Analysis Via Meta-learning.

Multi-Grained Fusion Network with Self-Distillation for Aspect-Based Multimodal Sentiment Analysis

Aspects Are Anchors: Towards Multimodal Aspect-based Sentiment Analysis Via Aspect-driven Alignment and Refinement

MIECF: Multi-faceted information extraction and cross-mixture fusion for multimodal aspect-based sentiment analysis

Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis

Prompt Fusion Interaction Transformer for Aspect-Based Multimodal Sentiment Analysis

MFSC: A Multimodal Aspect-Level Sentiment Classification Framework with Multi-Image Gate and Fusion Networks

CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Self-adaptive attention fusion for multimodal aspect-based sentiment analysis

Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis

M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis

CLMLF: A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection

Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt

A Contrastive Cross-Channel Data Augmentation Framework for Aspect-based Sentiment Analysis

Dual-Perspective Fusion Network for Aspect-Based Multimodal Sentiment Analysis