Abstract:In recent years, the spread of rumors on social media has intensified, which will produce a serious impact on the economy and society. Therefore, effective rumor detection has become a hot topic of concern. With the increasing complexity of neural network, the performance of models that only relies on single modal features are approaching the bottleneck, so researchers tried to encode multi-modal features. However, most of them only apply concatenation, similarity calculation, or attention based mechanism to multi-modal features, which do not fully exploit the connection between modalities. In order to mine the relationship between multi-modal features and research its value for rumor detection, we propose a multi-modal fusion rumor detection framework named improved Bilinear Block-diagonal Multi-modal Fusion model (BBMF). The proposed framework firstly encodes the propagation structure features and text features through a novel designed pagerank-based bidirectional Graph Convolutional Network and a topic linguistic fusion network respectively, then a newly designed bilinear block-diagonal fusion network is used to fuse multi-modal features and generate joint representations for rumor classification task. Specially, for the propagation modal feature encoding, the pagerank-based bidirectional Graph Convolutional Network calculates the influence of each node by capturing the relative position information of the reply. For the text modal feature encoding, BBMF extracts the linguistic features and topic features through two pre-trained models respectively, and generate a topic-guided high-order text features through the topic linguistic fusion network. In multi-modal feature fusion module, bilinear block-diagonal fusion network reduces the redundancy of bilinear pooling by the chain map layers and block-diagonal tensors with sparsity constraints, and generate high performance jointly representation features. We perform experiments on three real-world datasets. the results show that the proposed framework BBMF achieves better performance than the state-of-the-art models. In addition, we conduct a series of additional experiments to verify the advantages and significance of each component in the framework.

Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection.

Interpretable Graph Neural Network for Social Media Rumor Detection

Graph Attention Network with Cross-Modal Interaction for Rumor Detection

Multi-level Interaction Network for Multi-Modal Rumor Detection.

Multi-modal Rumor Detection Via Knowledge-Aware Heterogeneous Graph Convolutional Networks

Multi-modal Fusion Network for Rumor Detection with Texts and Images

MGAT-ESM: Multi-channel Graph Attention Neural Network with Event-Sharing Module for Rumor Detection

Graph-aware multi-feature interacting network for explainable rumor detection on social network

A Rumor Detection Model Based on Graph Convolutional Networks and Multimodal Features

GMRD

BiMGCL: Rumor Detection Via Bi- Directional Multi-Level Graph Contrastive Learning

MDMN: Multi-task and Domain Adaptation Based Multi-modal Network for Early Rumor Detection

Rumor Detection Model Based on Multimodal Machine Learning

Unifying Multimodal Source and Propagation Graph for Rumour Detection on Social Media with Missing Features

Combining Temporal and Interactive Features for Rumor Detection: A Graph Neural Network Based Model

Multimodal Fusion Network with Contrary Latent Topic Memory for Rumor Detection

MMRDF: An improved multitask multi-modal rumor detection framework

Improved Bilinear Block-Diagonal Multi-Modal Fusion Model for Rumor Detection

Adaptive Multimodal Graph Integration Network for Multimodal Sentiment Analysis

Rumor detection on social media using hierarchically aggregated feature via graph neural networks

Rumor Detection Based on Knowledge Enhancement and Graph Attention Network