Abstract:In recent years, the spread of rumors on social media has intensified, which will produce a serious impact on the economy and society. Therefore, effective rumor detection has become a hot topic of concern. With the increasing complexity of neural network, the performance of models that only relies on single modal features are approaching the bottleneck, so researchers tried to encode multi-modal features. However, most of them only apply concatenation, similarity calculation, or attention based mechanism to multi-modal features, which do not fully exploit the connection between modalities. In order to mine the relationship between multi-modal features and research its value for rumor detection, we propose a multi-modal fusion rumor detection framework named improved Bilinear Block-diagonal Multi-modal Fusion model (BBMF). The proposed framework firstly encodes the propagation structure features and text features through a novel designed pagerank-based bidirectional Graph Convolutional Network and a topic linguistic fusion network respectively, then a newly designed bilinear block-diagonal fusion network is used to fuse multi-modal features and generate joint representations for rumor classification task. Specially, for the propagation modal feature encoding, the pagerank-based bidirectional Graph Convolutional Network calculates the influence of each node by capturing the relative position information of the reply. For the text modal feature encoding, BBMF extracts the linguistic features and topic features through two pre-trained models respectively, and generate a topic-guided high-order text features through the topic linguistic fusion network. In multi-modal feature fusion module, bilinear block-diagonal fusion network reduces the redundancy of bilinear pooling by the chain map layers and block-diagonal tensors with sparsity constraints, and generate high performance jointly representation features. We perform experiments on three real-world datasets. the results show that the proposed framework BBMF achieves better performance than the state-of-the-art models. In addition, we conduct a series of additional experiments to verify the advantages and significance of each component in the framework.

Multi-level Interaction Network for Multi-Modal Rumor Detection.

Multimodal Fusion Network with Contrary Latent Topic Memory for Rumor Detection

Interpretable Graph Neural Network for Social Media Rumor Detection

Multi-modal Fusion Network for Rumor Detection with Texts and Images

MDMN: Multi-task and Domain Adaptation Based Multi-modal Network for Early Rumor Detection

MMRDF: An improved multitask multi-modal rumor detection framework

Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection.

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

Multi-Modal Meta Multi-Task Learning for Social Media Rumor Detection

Multimodal Rumor Detection by Using Additive Angular Margin with Class-Aware Attention for Hard Samples.

Graph Attention Network with Cross-Modal Interaction for Rumor Detection

Deep visual-linguistic fusion network considering cross-modal inconsistency for rumor detection

Multi-modal Rumor Detection Via Knowledge-Aware Heterogeneous Graph Convolutional Networks

A Syntactic Multi-Level Interaction Network for Rumor Detection

Deciphering Rumors: A Multi-Task Learning Approach with Intent-aware Hierarchical Contrastive Learning

MRML: Multimodal Rumor Detection by Deep Metric Learning

Multi-modal Rumor Detection on Modality Alignment and Multi-perspective Structures.

Graph-aware multi-feature interacting network for explainable rumor detection on social network

MSRD: Multi-Modal Web Rumor Detection Method

Focusing on Relevant Responses for Multi-modal Rumor Detection

Improved Bilinear Block-Diagonal Multi-Modal Fusion Model for Rumor Detection