Abstract:With the development of the Internet, users can freely publish posts on various social media platforms, which offers great convenience for keeping abreast of the world. However, posts usually carry many rumors, which require plenty of manpower for monitoring. Owing to the success of modern machine learning techniques, especially deep learning models, we tried to detect rumors as a classification problem automatically. Early attempts have always focused on building classifiers relying on image or text information, i.e., single modality in posts. Thereafter, several multimodal detection approaches employ an early or late fusion operator for aggregating multiple source information. Nevertheless, they only take advantage of multimodal embeddings for fusion and ignore another important detection factor, i.e., the intermodal inconsistency between modalities. To solve this problem, we develop a novel deep visual-linguistic fusion network (DVLFN) considering cross-modal inconsistency, which detects rumors by comprehensively considering modal aggregation and contrast information. Specifically, the DVLFN first utilizes visual and textual deep encoders, i.e., Faster R-CNN and bidirectional encoder representations from transformers, to extract global and regional embeddings for image and text modalities. Then, it predicts posts' authenticity from two aspects: (1) intermodal inconsistency, which employs the Wasserstein distance to efficiently measure the similarity between regional embeddings of different modalities, and (2) modal aggregation, which experimentally employs the early fusion to aggregate two modal embeddings for prediction. Consequently, the DVLFN can compose the final prediction based on the modal fusion and inconsistency measure. Experiments are conducted on three real-world multimedia rumor detection datasets collected from Reddit, GoodNews, and Weibo. The results validate the superior performance of the proposed DVLFN.

MR2: A Benchmark for Multimodal Retrieval-Augmented Rumor Detection in Social Media.

Multimodal Fusion Network with Contrary Latent Topic Memory for Rumor Detection

Detecting the Rumor Patterns Integrating Features of User, Content, and the Spreading Structure.

Rumor detection on social media with multimodal feature fusion

Focusing on Relevant Responses for Multi-modal Rumor Detection

Multimodal Short Video Rumor Detection System Based on Contrastive Learning

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

Automatic Rumor Detection on Microblogs: A Survey

Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection

Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs

Detect Rumors Using Time Series of Social Context Information on Microblogging Websites.

Calling to CNN-LSTM for Rumor Detection: A Deep Multi-channel Model for Message Veracity Classification in Microblogs

Dual Co-Attention-Based Multi-Feature Fusion Method for Rumor Detection

Unifying Multimodal Source and Propagation Graph for Rumour Detection on Social Media with Missing Features

CED: Credible Early Detection of Social Media Rumors

Rumor Detection on Social Media with Event Augmentations

Rumor2vec: A rumor detection framework with joint text and propagation structure representation learning

A Unified Contrastive Transfer Framework with Propagation Structure for Boosting Low-Resource Rumor Detection

Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learning

Rumor Detection with Self-supervised Learning on Texts and Social Graph

A Rumor Detection Method Based on Multimodal Feature Fusion by a Joining Aggregation Structure