Abstract:In recent years, the spread of rumors on social media has intensified, which will produce a serious impact on the economy and society. Therefore, effective rumor detection has become a hot topic of concern. With the increasing complexity of neural network, the performance of models that only relies on single modal features are approaching the bottleneck, so researchers tried to encode multi-modal features. However, most of them only apply concatenation, similarity calculation, or attention based mechanism to multi-modal features, which do not fully exploit the connection between modalities. In order to mine the relationship between multi-modal features and research its value for rumor detection, we propose a multi-modal fusion rumor detection framework named improved Bilinear Block-diagonal Multi-modal Fusion model (BBMF). The proposed framework firstly encodes the propagation structure features and text features through a novel designed pagerank-based bidirectional Graph Convolutional Network and a topic linguistic fusion network respectively, then a newly designed bilinear block-diagonal fusion network is used to fuse multi-modal features and generate joint representations for rumor classification task. Specially, for the propagation modal feature encoding, the pagerank-based bidirectional Graph Convolutional Network calculates the influence of each node by capturing the relative position information of the reply. For the text modal feature encoding, BBMF extracts the linguistic features and topic features through two pre-trained models respectively, and generate a topic-guided high-order text features through the topic linguistic fusion network. In multi-modal feature fusion module, bilinear block-diagonal fusion network reduces the redundancy of bilinear pooling by the chain map layers and block-diagonal tensors with sparsity constraints, and generate high performance jointly representation features. We perform experiments on three real-world datasets. the results show that the proposed framework BBMF achieves better performance than the state-of-the-art models. In addition, we conduct a series of additional experiments to verify the advantages and significance of each component in the framework.

Multimodal Fusion Network with Contrary Latent Topic Memory for Rumor Detection

Multimodal Rumor Detection by Using Additive Angular Margin with Class-Aware Attention for Hard Samples.

Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs

Multimodal Fusion with Dual-Attention Based on Textual Double-Embedding Networks for Rumor Detection

Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection

Rumor detection on social media with multimodal feature fusion

MSRD: Multi-Modal Web Rumor Detection Method

MRML: Multimodal Rumor Detection by Deep Metric Learning

Improved Bilinear Block-Diagonal Multi-Modal Fusion Model for Rumor Detection

Multi-level Interaction Network for Multi-Modal Rumor Detection.

Rumor Detection Model Based on Multimodal Machine Learning

Multi-modal Fusion Network for Rumor Detection with Texts and Images

A Rumor Detection Method Based on Multimodal Feature Fusion by a Joining Aggregation Structure

MDMN: Multi-task and Domain Adaptation Based Multi-modal Network for Early Rumor Detection

MMRDF: An improved multitask multi-modal rumor detection framework

Cross-Modal Adversarial Contrastive Learning for Multi-Modal Rumor Detection

MVACLNet: A Multimodal Virtual Augmentation Contrastive Learning Network for Rumor Detection

Multimodal Short Video Rumor Detection System Based on Contrastive Learning

Multi-modal Knowledge-aware Event Memory Network for Social Media Rumor Detection

Multi-view Learning with Distinguishable Feature Fusion for Rumor Detection

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection