Abstract:Identifying sarcastic clues from both textual and visual information has become an important research issue, called Multimodal Sarcasm Detection. In this article, we investigate multimodal sarcasm detection from a novel perspective, where a multimodal graph contrastive learning strategy is proposed to fuse and distinguish the sarcastic clues for textual modality and visual modality. Specifically, we first utilize object detection to derive the crucial visual regions accompanied by their captions of the images, which allows better learning of the key visual regions of visual modality. In addition, to make full use of the semantic information of the visual modality, we employ optical character recognition to extract the textual content in the images. Then, based on image regions, the textual content of visual modality, and the context of the textual modality, we build a multimodal graph for each sample to model the intricate sarcastic relations between modalities. Furthermore, we devise a graph-oriented contrastive learning strategy to leverage the correlations in the same label and differences between different labels, so as to capture better multimodal representations for multimodal sarcasm detection. Extensive experiments show that our method outperforms the previous best baseline models (with a 2.47% improvement in Accuracy, a 1.99% improvement in F-score, and a 2.20% improvement in Macro F-score). The ablation study shows that both multimodal graph structure and graph-oriented contrastive learning are important to our framework. Further, the experiments of using different pre-trained methods show that the proposed multimodal graph contrastive learning framework can directly work with various pre-trained models and achieve outstanding performance in multimodal sarcasm detection.

An attention-based, context-aware multimodal fusion method for sarcasm detection using inter-modality inconsistency

Multi-Modal Sarcasm Detection with Sentiment Word Embedding

Multi-modal sarcasm detection based on emotion perception and cross-modality attention fusion

Attention-based multi-modal fusion sarcasm detection

Sarcasm driven by sentiment: A sentiment-aware hierarchical fusion network for multimodal sarcasm detection

Modeling Incongruity Between Modalities for Multimodal Sarcasm Detection

Dual-level adaptive incongruity-enhanced model for multimodal sarcasm detection

Sarcasm Detection of Dual Multimodal Contrastive Attention Networks.

Enhanced Semantic Representation Learning for Sarcasm Detection by Integrating Context-Aware Attention and Fusion Network

Learning Multi-Task Commonness and Uniqueness for Multi-Modal Sarcasm Detection and Sentiment Analysis in Conversation

Multi-Modal Sarcasm Detection Based on Contrastive Attention Mechanism

A Semantic Enhancement Framework for Multimodal Sarcasm Detection

AMuSeD: An Attentive Deep Neural Network for Multimodal Sarcasm Detection Incorporating Bi-modal Data Augmentation

RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detection

Sarcasm Detection Method Based on Fusion of Text Semantics and Social Behavior Information

MMSD-CAF: MultiModal Sarcasm Detection using CoAttention and Fusion Mechanisms

Towards Multi-Modal Sarcasm Detection via Hierarchical Congruity Modeling with Knowledge Enhancement

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Fusion and Discrimination: A Multimodal Graph Contrastive Learning Framework for Multimodal Sarcasm Detection

Multi-Modal Sarcasm Detection In Twitter With Hierarchical Fusion Model

MIAN: Multi-head Incongruity Aware Attention Network with Transfer Learning for Sarcasm Detection