Abstract:Identifying sarcastic clues from both textual and visual information has become an important research issue, called Multimodal Sarcasm Detection. In this article, we investigate multimodal sarcasm detection from a novel perspective, where a multimodal graph contrastive learning strategy is proposed to fuse and distinguish the sarcastic clues for textual modality and visual modality. Specifically, we first utilize object detection to derive the crucial visual regions accompanied by their captions of the images, which allows better learning of the key visual regions of visual modality. In addition, to make full use of the semantic information of the visual modality, we employ optical character recognition to extract the textual content in the images. Then, based on image regions, the textual content of visual modality, and the context of the textual modality, we build a multimodal graph for each sample to model the intricate sarcastic relations between modalities. Furthermore, we devise a graph-oriented contrastive learning strategy to leverage the correlations in the same label and differences between different labels, so as to capture better multimodal representations for multimodal sarcasm detection. Extensive experiments show that our method outperforms the previous best baseline models (with a 2.47% improvement in Accuracy, a 1.99% improvement in F-score, and a 2.20% improvement in Macro F-score). The ablation study shows that both multimodal graph structure and graph-oriented contrastive learning are important to our framework. Further, the experiments of using different pre-trained methods show that the proposed multimodal graph contrastive learning framework can directly work with various pre-trained models and achieve outstanding performance in multimodal sarcasm detection.

Towards Multimodal Sarcasm Detection Via Label-Aware Graph Contrastive Learning with Back-Translation Augmentation

Fusion and Discrimination: A Multimodal Graph Contrastive Learning Framework for Multimodal Sarcasm Detection

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Multi-perspective Contrastive Learning Framework Guided by Sememe Knowledge and Label Information for Sarcasm Detection

Dual-level adaptive incongruity-enhanced model for multimodal sarcasm detection

Learning Multi-Task Commonness and Uniqueness for Multi-Modal Sarcasm Detection and Sentiment Analysis in Conversation

Enhancing Cross-Lingual Sarcasm Detection by a Prompt Learning Framework with Data Augmentation and Contrastive Learning

Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism

Multi-View Incongruity Learning for Multimodal Sarcasm Detection

A Semantic Enhancement Framework for Multimodal Sarcasm Detection

Sarcasm driven by sentiment: A sentiment-aware hierarchical fusion network for multimodal sarcasm detection

Mutual-Enhanced Incongruity Learning Network for Multi-Modal Sarcasm Detection

Multimodal Sarcasm Detection via Hybrid Classifier with Optimistic Logic

An attention-based, context-aware multimodal fusion method for sarcasm detection using inter-modality inconsistency

MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System

KnowleNet: Knowledge fusion network for multimodal sarcasm detection

Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Attention-based multi-modal fusion sarcasm detection

CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)