Similarity-Aware Multimodal Prompt Learning for Fake News Detection

Ye Jiang,Xiaomin Yu,Yimin Wang,Xiaoman Xu,Xingyi Song,Diana Maynard

DOI: https://doi.org/10.1016/j.ins.2023.119446

2023-06-16

Abstract:The standard paradigm for fake news detection mainly utilizes text information to model the truthfulness of news. However, the discourse of online fake news is typically subtle and it requires expert knowledge to use textual information to debunk fake news. Recently, studies focusing on multimodal fake news detection have outperformed text-only methods. Recent approaches utilizing the pre-trained model to extract unimodal features, or fine-tuning the pre-trained model directly, have become a new paradigm for detecting fake news. Again, this paradigm either requires a large number of training instances, or updates the entire set of pre-trained model parameters, making real-world fake news detection impractical. Furthermore, traditional multimodal methods fuse the cross-modal features directly without considering that the uncorrelated semantic representation might inject noise into the multimodal features. This paper proposes a Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. First, we incorporate prompt learning into multimodal fake news detection. Prompt learning, which only tunes prompts with a frozen language model, can reduce memory usage significantly and achieve comparable performances, compared with fine-tuning. We analyse three prompt templates with a soft verbalizer to detect fake news. In addition, we introduce the similarity-aware fusing method to adaptively fuse the intensity of multimodal representation and mitigate the noise injection via uncorrelated cross-modal features. For evaluation, SAMPLE surpasses the F1 and the accuracies of previous works on two benchmark multimodal datasets, demonstrating the effectiveness of the proposed method in detecting fake news. In addition, SAMPLE also is superior to other approaches regardless of few-shot and data-rich settings.

Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the following issues:

1. **Multimodal Information Fusion in Fake News Detection**:
- Traditional multimodal fake news detection methods directly fuse cross-modal features but do not consider the potential semantic irrelevance between different modalities, which may lead to noise introduction.

2. **Effective Detection with Limited Labeled Data**:
- Existing deep learning methods usually require a large amount of labeled data to achieve good performance. However, in practical applications, manually labeled fake news data is scarce.

3. **Application of Prompt Learning in Multimodal Fake News Detection**:
- Prompt Learning can leverage the knowledge of pre-trained models and achieve good results even with a small amount of labeled data. However, current prompt-based learning methods mainly focus on textual information and fail to fully utilize image information.

To address these issues, the authors propose a framework named **Similarity-Aware Multimodal Prompt Learning (SAMPLE)**. This framework combines prompt learning and multimodal feature fusion techniques by calculating the semantic similarity between text and images to adjust the intensity of the fused multimodal representation, thereby reducing noise introduction and improving the accuracy of fake news detection. Experimental results show that SAMPLE performs excellently on two benchmark multimodal datasets (PolitiFact and GossipCop), significantly outperforming existing methods in both data-rich and limited data scenarios.