CFEVER: A Chinese Fact Extraction and VERification Dataset

Ying-Jia Lin,Chun-Yi Lin,Chia-Jen Yeh,Yi-Ting Li,Yun-Yu Hu,Chih-Hao Hsu,Mei-Feng Lee,Hung-Yu Kao

2024-02-20

Abstract:We present CFEVER, a Chinese dataset designed for Fact Extraction and VERification. CFEVER comprises 30,012 manually created claims based on content in Chinese Wikipedia. Each claim in CFEVER is labeled as "Supports", "Refutes", or "Not Enough Info" to depict its degree of factualness. Similar to the FEVER dataset, claims in the "Supports" and "Refutes" categories are also annotated with corresponding evidence sentences sourced from single or multiple pages in Chinese Wikipedia. Our labeled dataset holds a Fleiss' kappa value of 0.7934 for five-way inter-annotator agreement. In addition, through the experiments with the state-of-the-art approaches developed on the FEVER dataset and a simple baseline for CFEVER, we demonstrate that our dataset is a new rigorous benchmark for factual extraction and verification, which can be further used for developing automated systems to alleviate human fact-checking efforts. CFEVER is available at https://ikmlab.github.io/CFEVER.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the problem of Chinese fact verification. Specifically, the authors propose the CFEVER dataset, which is currently the largest Chinese fact extraction and verification dataset. CFEVER contains 30,012 manually created statements based on Chinese Wikipedia content and labels these statements as "support," "refute," or "not enough information." Similar to existing English fact verification datasets, the "support" and "refute" categories in CFEVER also annotate relevant evidence sentences from single or multiple Chinese Wikipedia pages.

The paper emphasizes the importance of fact verification, especially in the context of the rapid spread of misinformation due to the prevalence of modern media platforms. However, existing fact verification systems are mostly based on English datasets, and Chinese text is often more ambiguous and difficult to identify, necessitating a high-quality Chinese dataset to support automated fact verification systems. By constructing the CFEVER dataset, the authors hope to provide a benchmark for Chinese fact verification to promote related research development and reduce the workload of manual fact-checking. Additionally, the paper demonstrates the performance of existing state-of-the-art methods on CFEVER through experiments, indicating that this dataset is a challenging new benchmark.