Abstract:Multimedia content has become ubiquitous on social media platforms, leading to the rise of multimodal misinformation (MM) and the urgent need for effective strategies to detect and prevent its spread. In recent years, the challenge of multimodal misinformation detection (MMD) has garnered significant attention by researchers and has mainly involved the creation of annotated, weakly annotated, or synthetically generated training datasets, along with the development of various deep learning MMD models. However, the problem of unimodal bias has been overlooked, where specific patterns and biases in MMD benchmarks can result in biased or unimodal models outperforming their multimodal counterparts on an inherently multimodal task; making it difficult to assess progress. In this study, we systematically investigate and identify the presence of unimodal bias in widely-used MMD benchmarks, namely VMU-Twitter and COSMOS. To address this issue, we introduce the "VERification of Image-TExt pairs" (VERITE) benchmark for MMD which incorporates real-world data, excludes "asymmetric multimodal misinformation" and utilizes "modality balancing". We conduct an extensive comparative study with a Transformer-based architecture that shows the ability of VERITE to effectively address unimodal bias, rendering it a robust evaluation framework for MMD. Furthermore, we introduce a new method -- termed Crossmodal HArd Synthetic MisAlignment (CHASMA) -- for generating realistic synthetic training data that preserve crossmodal relations between legitimate images and false human-written captions. By leveraging CHASMA in the training process, we observe consistent and notable improvements in predictive performance on VERITE; with a 9.2% increase in accuracy. We release our code at: <a class="link-external link-https" href="https://github.com/stevejpapad/image-text-verification" rel="external noopener nofollow">this https URL</a>

Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation

Synthetic Misinformers: Generating and Combating Multimodal Misinformation

Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMs

Detecting Out-of-Context Image-Caption Pairs in News: A Counter-Intuitive Method

Similarity over Factuality: Are we making progress on multimodal out-of-context misinformation detection?

ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms

Interpretable Multimodal Out-of-context Detection with Soft Logic Regularization

Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalisation of Misinformation Detection Models

MAD-Sherlock: Multi-Agent Debates for Out-of-Context Misinformation Detection

Misinformation Detection in Social Media Video Posts

SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection

Multimodal Detection of Information Disorder from Social Media

An image and text-based multimodal model for detecting fake news in OSN's

Detecting Misinformation in Multimedia Content through Cross-Modal Entity Consistency: A Dual Learning Approach

Text-Image De-Contextualization Detection Using Vision-Language Models

"Image, Tell me your story!" Predicting the original meta-context of visual misinformation

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias

Identifying multimodal misinformation leveraging novelty detection and emotion recognition

Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation