Abstract:We present an overview of the second edition of the CheckThat! Lab at CLEF 2019. The lab featured two tasks in two different languages: English and Arabic. Task 1 (English) challenged the participating systems to predict which claims in a political debate or speech should be prioritized for fact-checking. Task 2 (Arabic) asked to (A) rank a given set of Web pages with respect to a check-worthy claim based on their usefulness for fact-checking that claim, (B) classify these same Web pages according to their degree of usefulness for fact-checking the target claim, (C) identify useful passages from these pages, and (D) use the useful pages to predict the claim's factuality. CheckThat! provided a full evaluation framework, consisting of data in English (derived from fact-checking sources) and Arabic (gathered and annotated from scratch) and evaluation based on mean average precision (MAP) and normalized discounted cumulative gain (nDCG) for ranking, and F1 for classification. A total of 47 teams registered to participate in this lab, and fourteen of them actually submitted runs (compared to nine last year). The evaluation results show that the most successful approaches to Task 1 used various neural networks and logistic regression. As for Task 2, learning-to-rank was used by the highest scoring runs for subtask A, while different classifiers were used in the other subtasks. We release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in the important tasks of check-worthiness estimation and automatic claim verification.

Overview of the CLEF--2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News

CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media

Overview of the CLEF-2019 CheckThat!: Automatic Identification and Verification of Claims

Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media

Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models

Team Alex at CLEF CheckThat! 2020: Identifying Check-Worthy Tweets With Transformer Models

Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence

Overview of the CLEF eHealth Evaluation Lab 2019

FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning

Overview of the CLEF eHealth Evaluation Lab 2015

FacTeR-Check: Semi-automated fact-checking through Semantic Similarity and Natural Language Inference

SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Automated Fact-Checking for Assisting Human Fact-Checkers

Check_square at CheckThat! 2020: Claim Detection in Social Media via Fusion of Transformer and Syntactic Features

The Automated Verification of Textual Claims (AVeriTeC) Shared Task

Findings of Factify 2: Multimodal Fake News Detection

Fraunhofer SIT at CheckThat! 2023: Mixing Single-Modal Classifiers to Estimate the Check-Worthiness of Multi-Modal Tweets

Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications

That is a Known Lie: Detecting Previously Fact-Checked Claims

Overview of the CLAIMSCAN-2023: Uncovering Truth in Social Media through Claim Detection and Identification of Claim Spans