Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection

Beizhe Hu,Qiang Sheng,Juan Cao,Yuhui Shi,Yang Li,Danding Wang,Peng Qi

DOI: https://doi.org/10.1609/aaai.v38i20.30214

2024-01-22

Abstract:Detecting fake news requires both a delicate sense of diverse clues and a profound understanding of the real-world background, which remains challenging for detectors based on small language models (SLMs) due to their knowledge and capability limitations. Recent advances in large language models (LLMs) have shown remarkable performance in various tasks, but whether and how LLMs could help with fake news detection remains underexplored. In this paper, we investigate the potential of LLMs in fake news detection. First, we conduct an empirical study and find that a sophisticated LLM such as GPT 3.5 could generally expose fake news and provide desirable multi-perspective rationales but still underperforms the basic SLM, fine-tuned BERT. Our subsequent analysis attributes such a gap to the LLM's inability to select and integrate rationales properly to conclude. Based on these findings, we propose that current LLMs may not substitute fine-tuned SLMs in fake news detection but can be a good advisor for SLMs by providing multi-perspective instructive rationales. To instantiate this proposal, we design an adaptive rationale guidance network for fake news detection (ARG), in which SLMs selectively acquire insights on news analysis from the LLMs' rationales. We further derive a rationale-free version of ARG by distillation, namely ARG-D, which services cost-sensitive scenarios without querying LLMs. Experiments on two real-world datasets demonstrate that ARG and ARG-D outperform three types of baseline methods, including SLM-based, LLM-based, and combinations of small and large language models.

Computation and Language,Artificial Intelligence,Computers and Society

What problem does this paper attempt to address?

The paper attempts to address the issue of the effective application of large language models (LLMs) in fake news detection. Specifically, the authors explore the potential role of LLMs in fake news detection and try to find out how to leverage LLMs to improve the effectiveness of fake news detection. The study finds that although LLMs can provide reasonable explanatory bases from multiple perspectives, they perform worse than fine-tuned small language models (SLMs) in judging the authenticity of news. Based on this finding, the authors propose a new method, the Adaptive Rationale Guidance Network (ARG), which combines the advantages of SLMs and LLMs to improve fake news detection performance.

The main objectives include:
1. Exploring the role of LLMs in fake news detection, especially their potential in providing multi-perspective explanations.
2. Designing a novel ARG network and its distilled version ARG-D to compensate for the shortcomings of both SLMs and LLMs.
3. Proposing a solution for cost-sensitive scenarios that can utilize the information provided by LLMs even without directly querying them.