Abstract:With the development of the Internet, users can freely publish posts on various social media platforms, which offers great convenience for keeping abreast of the world. However, posts usually carry many rumors, which require plenty of manpower for monitoring. Owing to the success of modern machine learning techniques, especially deep learning models, we tried to detect rumors as a classification problem automatically. Early attempts have always focused on building classifiers relying on image or text information, i.e., single modality in posts. Thereafter, several multimodal detection approaches employ an early or late fusion operator for aggregating multiple source information. Nevertheless, they only take advantage of multimodal embeddings for fusion and ignore another important detection factor, i.e., the intermodal inconsistency between modalities. To solve this problem, we develop a novel deep visual-linguistic fusion network (DVLFN) considering cross-modal inconsistency, which detects rumors by comprehensively considering modal aggregation and contrast information. Specifically, the DVLFN first utilizes visual and textual deep encoders, i.e., Faster R-CNN and bidirectional encoder representations from transformers, to extract global and regional embeddings for image and text modalities. Then, it predicts posts' authenticity from two aspects: (1) intermodal inconsistency, which employs the Wasserstein distance to efficiently measure the similarity between regional embeddings of different modalities, and (2) modal aggregation, which experimentally employs the early fusion to aggregate two modal embeddings for prediction. Consequently, the DVLFN can compose the final prediction based on the modal fusion and inconsistency measure. Experiments are conducted on three real-world multimedia rumor detection datasets collected from Reddit, GoodNews, and Weibo. The results validate the superior performance of the proposed DVLFN.

Multimodal Rumor Detection by Using Additive Angular Margin with Class-Aware Attention for Hard Samples.

Multimodal Fusion Network with Contrary Latent Topic Memory for Rumor Detection

MRML: Multimodal Rumor Detection by Deep Metric Learning

Cross-Modal Adversarial Contrastive Learning for Multi-Modal Rumor Detection

MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection

MMRDF: An improved multitask multi-modal rumor detection framework

Rumor Detection Model Based on Multimodal Machine Learning

Multimodal Fusion with Dual-Attention Based on Textual Double-Embedding Networks for Rumor Detection

MDMN: Multi-task and Domain Adaptation Based Multi-modal Network for Early Rumor Detection

Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection

MSRD: Multi-Modal Web Rumor Detection Method

A Rumor Detection Method Based on Multimodal Feature Fusion by a Joining Aggregation Structure

Focusing on Relevant Responses for Multi-modal Rumor Detection

Rumor detection on social media with multimodal feature fusion

Multimodal Short Video Rumor Detection System Based on Contrastive Learning

Multi-level Interaction Network for Multi-Modal Rumor Detection.

Multi-view Learning with Distinguishable Feature Fusion for Rumor Detection

Multi-modal Rumour Detection Using Bilinear Pooling and Domain Adversarial Neural Networks.

Multimodal Disentangled Domain Adaption for Social Media Event Rumor Detection

Leveraging Contrastive Learning and Knowledge Distillation for Incomplete Modality Rumor Detection

Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs