FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs

Eun Cheol Choi,Emilio Ferrara

2024-02-09

Abstract:Our society is facing rampant misinformation harming public health and trust. To address the societal challenge, we introduce FACT-GPT, a system leveraging Large Language Models (LLMs) to automate the claim matching stage of fact-checking. FACT-GPT, trained on a synthetic dataset, identifies social media content that aligns with, contradicts, or is irrelevant to previously debunked claims. Our evaluation shows that our specialized LLMs can match the accuracy of larger models in identifying related claims, closely mirroring human judgment. This research provides an automated solution for efficient claim matching, demonstrates the potential of LLMs in supporting fact-checkers, and offers valuable resources for further research in the field.

Human-Computer Interaction,Social and Information Networks,Computers and Society

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to quickly identify and match information that has been verified as false (i.e., false claims) on social media, in order to support fact - checking work. Specifically, the paper introduces a system named FACT - GPT, which utilizes large language models (LLMs) to automate the "claim - matching" stage in the fact - checking process. By training these models to recognize social media content that is consistent with, contradictory to, or irrelevant to previously debunked false claims, FACT - GPT aims to improve the efficiency of fact - checking, reduce the workload of repetitive verification, and support content moderation on online platforms. The core objective of the paper is to demonstrate the capabilities of large language models in assisting fact - checkers, especially their potential in detecting rumors and reducing the spread of false information. In addition, the research also explores how to generate synthetic training datasets to optimize the performance of these models and evaluates the performance differences of different models on this task. In this way, the paper not only provides technical solutions but also offers valuable resources for future research.

FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMs

Automated Claim Matching with Large Language Models: Empowering Fact-Checkers in the Fight Against Misinformation

The Perils & Promises of Fact-checking with Large Language Models

The perils and promises of fact-checking with large language models

FactLLaMA: Optimizing Instruction-Following Language Models with External Knowledge for Automated Fact-Checking

Factuality challenges in the era of large language models and opportunities for fact-checking

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Self-Checker: Plug-and-Play Modules for Fact-Checking with Large Language Models

OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs

FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning

Multimodal Large Language Models to Support Real-World Fact-Checking

Generative Large Language Models in Automated Fact-Checking: A Survey

Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification

OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

FactAlign: Long-form Factuality Alignment of Large Language Models

Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

The Earth is Flat? Unveiling Factual Errors in Large Language Models

Long-form factuality in large language models

Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?