Scaling up fact-checking using the wisdom of crowds

Jennifer Allen,Antonio A. Arechar,Gordon Pennycook,David G. Rand

DOI: https://doi.org/10.1126/sciadv.abf4393

IF: 13.6

2021-09-03

Science Advances

Abstract:When rating articles’ accuracy, a small politically balanced crowd of laypeople yields high agreement with fact-checkers.

multidisciplinary sciences

What problem does this paper attempt to address?

Fact-Checking at Scale with DimensionRank

Gregory Coppola

DOI: https://doi.org/10.48550/arXiv.2010.10685

2020-10-21

Information Retrieval

Abstract:The most important problem that has emerged after twenty years of popular internet usage is that of fact-checking at scale. This problem is experienced acutely in both of the major internet application platform types, web search and social media. We offer a working definition of what a "platform" is. We critically deconstruct what we call the "PolitiFact" model of fact checking, and show it to be inherently inferior for fact-checking at scale to a platform-b ased solution. Our central contribution is to show how to effectively platformize the problem of fact-checking at scale. We show how a two-dimensional rating system, with dimensions agreement and hotness allows us to create information-seeking queries not possible with the on e-dimensional rating system predominating on existing platforms. And, we show that, underlying our user-friendly user-interface, lies a system that allows the creation of formal proofs in the propositional calculus. Our algorithm is implemented in our open-source DimensionRank software package available at "https://thinkdifferentagain.art".
The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale

Michael Soprano,Kevin Roitero,David La Barbera,Davide Ceolin,Damiano Spina,Stefano Mizzaro,Gianluca Demartini

DOI: https://doi.org/10.1016/j.ipm.2021.102710

2021-08-23

Abstract:Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased ("cherrypicked"), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker's Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.

Information Retrieval
Searching For or Reviewing Evidence Improves Crowdworkers' Misinformation Judgments and Reduces Partisan Bias

Paul Resnick,Aljohara Alfayez,Jane Im,Eric Gilbert

DOI: https://doi.org/10.1177/26339137231173407

2023-04-11

Abstract:Can crowd workers be trusted to judge whether news-like articles circulating on the Internet are misleading, or does partisanship and inexperience get in the way? And can the task be structured in a way that reduces partisanship? We assembled pools of both liberal and conservative crowd raters and tested three ways of asking them to make judgments about 374 articles. In a no research condition, they were just asked to view the article and then render a judgment. In an individual research condition, they were also asked to search for corroborating evidence and provide a link to the best evidence they found. In a collective research condition, they were not asked to search, but instead to review links collected from workers in the individual research condition. Both research conditions reduced partisan disagreement in judgments. The individual research condition was most effective at producing alignment with journalists' assessments. In this condition, the judgments of a panel of sixteen or more crowd workers were better than that of a panel of three expert journalists, as measured by alignment with a held out journalist's ratings.

Human-Computer Interaction
Who Checks the Checkers? Exploring Source Credibility in Twitter's Community Notes

Uku Kangur,Roshni Chakraborty,Rajesh Sharma

2024-06-18

Abstract:In recent years, the proliferation of misinformation on social media platforms has become a significant concern. Initially designed for sharing information and fostering social connections, platforms like Twitter (now rebranded as X) have also unfortunately become conduits for spreading misinformation. To mitigate this, these platforms have implemented various mechanisms, including the recent suggestion to use crowd-sourced non-expert fact-checkers to enhance the scalability and efficiency of content vetting. An example of this is the introduction of Community Notes on Twitter. While previous research has extensively explored various aspects of Twitter tweets, such as information diffusion, sentiment analytics and opinion summarization, there has been a limited focus on the specific feature of Twitter Community Notes, despite its potential role in crowd-sourced fact-checking. Prior research on Twitter Community Notes has involved empirical analysis of the feature's dataset and comparative studies that also include other methods like expert fact-checking. Distinguishing itself from prior works, our study covers a multi-faceted analysis of sources and audience perception within Community Notes. We find that the majority of cited sources are news outlets that are left-leaning and are of high factuality, pointing to a potential bias in the platform's community fact-checking. Left biased and low factuality sources validate tweets more, while Center sources are used more often to refute tweet content. Additionally, source factuality significantly influences public agreement and helpfulness of the notes, highlighting the effectiveness of the Community Notes Ranking algorithm. These findings showcase the impact and biases inherent in community-based fact-checking initiatives.

Computers and Society,Social and Information Networks
Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence

Gengyu Wang,Kate Harwood,Lawrence Chillrud,Amith Ananthram,Melanie Subbiah,Kathleen McKeown

2023-05-30

Abstract:We present a new fact-checking benchmark, Check-COVID, that requires systems to verify claims about COVID-19 from news using evidence from scientific articles. This approach to fact-checking is particularly challenging as it requires checking internet text written in everyday language against evidence from journal articles written in formal academic language. Check-COVID contains 1, 504 expert-annotated news claims about the coronavirus paired with sentence-level evidence from scientific journal articles and veracity labels. It includes both extracted (journalist-written) and composed (annotator-written) claims. Experiments using both a fact-checking specific system and GPT-3.5, which respectively achieve F1 scores of 76.99 and 69.90 on this task, reveal the difficulty of automatically fact-checking both claim types and the importance of in-domain data for good performance. Our data and models are released publicly at <a class="link-external link-https" href="https://github.com/posuer/Check-COVID" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence,Computers and Society
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Yuxia Wang,Revanth Gangi Reddy,Zain Muhammad Mujahid,Arnav Arora,Aleksandr Rubashevskii,Jiahui Geng,Osama Mohammed Afzal,Liangming Pan,Nadav Borenstein,Aditya Pillai,Isabelle Augenstein,Iryna Gurevych,Preslav Nakov

DOI: https://doi.org/10.48550/arXiv.2311.09000

2024-04-16

Abstract:The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and <a class="link-external link-http" href="http://Perplexity.ai" rel="external noopener nofollow">this http URL</a> are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at <a class="link-external link-https" href="https://github.com/yuxiaw/Factcheck-GPT" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Some Observations on Fact-Checking Work with Implications for Computational Support

Rob Procter,Miguel Arana-Catania,Yulan He,Maria Liakata,Arkaitz Zubiaga,Elena Kochkina,Runcong Zhao

2023-07-06

Abstract:Social media and user-generated content (UGC) have become increasingly important features of journalistic work in a number of different ways. However, the growth of misinformation means that news organisations have had devote more and more resources to determining its veracity and to publishing corrections if it is found to be misleading. In this work, we present the results of interviews with eight members of fact-checking teams from two organisations. Team members described their fact-checking processes and the challenges they currently face in completing a fact-check in a robust and timely way. The former reveals, inter alia, significant differences in fact-checking practices and the role played by collaboration between team members. We conclude with a discussion of the implications for the development and application of computational tools, including where computational tool support is currently lacking and the importance of being able to accommodate different fact-checking practices.

Human-Computer Interaction
Checking the Fact-Checkers: the Role of Source Type, Perceived Credibility, and Individual Differences in Fact-Checking Effectiveness

Xingyu Liu,Li Qi,Laurent Wang,Miriam J. Metzger

DOI: https://doi.org/10.1177/00936502231206419

IF: 6.3

2023-01-01

Communication Research

Abstract:This study investigates fact-checking effectiveness in reducing belief in misinformation across various types of fact-check sources (i.e., professional fact-checkers, mainstream news outlets, social media platforms, artificial intelligence, and crowdsourcing). We examine fact-checker credibility perceptions as a mechanism to explain variance in fact-checking effectiveness across sources, while taking individual differences into account (i.e., analytic thinking and alignment with the fact-check verdict). An experiment with 859 participants revealed few differences in effectiveness across fact-checking sources but found that sources perceived as more credible are more effective. Indeed, the data show that perceived credibility of fact-check sources mediates the relationship between exposure to fact-checking messages and their effectiveness for some source types. Moreover, fact-checker credibility moderates the effect of alignment on effectiveness, while analytic thinking is unrelated to fact-checker credibility perceptions, alignment, and effectiveness. Other theoretical contributions include extending the scope of the credibility-persuasion association and the MAIN model to the fact-checking context, and empirically verifying a critical component of the two-step motivated reasoning model of misinformation correction.
HybridFC: A Hybrid Fact-Checking Approach for Knowledge Graphs

Umair Qudus,Michael Roeder,Muhammad Saleem,Axel-Cyrille Ngonga Ngomo

DOI: https://doi.org/10.1007/978-3-031-19433-7_27

2024-09-11

Abstract:We consider fact-checking approaches that aim to predict the veracity of assertions in knowledge graphs. Five main categories of fact-checking approaches for knowledge graphs have been proposed in the recent literature, of which each is subject to partially overlapping limitations. In particular, current text-based approaches are limited by manual feature engineering. Path-based and rule-based approaches are limited by their exclusive use of knowledge graphs as background knowledge, and embedding-based approaches suffer from low accuracy scores on current fact-checking tasks. We propose a hybrid approach -- dubbed HybridFC -- that exploits the diversity of existing categories of fact-checking approaches within an ensemble learning setting to achieve a significantly better prediction performance. In particular, our approach outperforms the state of the art by 0.14 to 0.27 in terms of Area Under the Receiver Operating Characteristic curve on the FactBench dataset. Our code is open-source and can be found at <a class="link-external link-https" href="https://github.com/dice-group/HybridFC" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Artificial Intelligence,Databases
The State of Human-centered NLP Technology for Fact-checking

Anubrata Das,Houjiang Liu,Venelin Kovatchev,Matthew Lease

DOI: https://doi.org/10.48550/arXiv.2301.03056

2023-01-08

Computation and Language

Abstract:Misinformation threatens modern society by promoting distrust in science, changing narratives in public health, heightening social polarization, and disrupting democratic elections and financial markets, among a myriad of other societal harms. To address this, a growing cadre of professional fact-checkers and journalists provide high-quality investigations into purported facts. However, these largely manual efforts have struggled to match the enormous scale of the problem. In response, a growing body of Natural Language Processing (NLP) technologies have been proposed for more scalable fact-checking. Despite tremendous growth in such research, however, practical adoption of NLP technologies for fact-checking still remains in its infancy today. In this work, we review the capabilities and limitations of the current NLP technologies for fact-checking. Our particular focus is to further chart the design space for how these technologies can be harnessed and refined in order to better meet the needs of human fact-checkers. To do so, we review key aspects of NLP-based fact-checking: task formulation, dataset construction, modeling, and human-centered strategies, such as explainable models and human-in-the-loop approaches. Next, we review the efficacy of applying NLP-based fact-checking tools to assist human fact-checkers. We recommend that future research include collaboration with fact-checker stakeholders early on in NLP research, as well as incorporation of human-centered design practices in model development, in order to further guide technology development for human use and practical adoption. Finally, we advocate for more research on benchmark development supporting extrinsic evaluation of human-centered fact-checking technologies.
Fact-checking information from large language models can decrease headline discernment

Matthew R. DeVerna,Harry Yaojun Yan,Kai-Cheng Yang,Filippo Menczer

2024-08-08

Abstract:Fact checking can be an effective strategy against misinformation, but its implementation at scale is impeded by the overwhelming volume of information online. Recent artificial intelligence (AI) language models have shown impressive ability in fact-checking tasks, but how humans interact with fact-checking information provided by these models is unclear. Here, we investigate the impact of fact-checking information generated by a popular large language model (LLM) on belief in, and sharing intent of, political news headlines in a preregistered randomized control experiment. Although the LLM accurately identifies most false headlines (90%), we find that this information does not significantly improve participants' ability to discern headline accuracy or share accurate news. In contrast, viewing human-generated fact checks enhances discernment in both cases. Subsequent analysis reveals that the AI fact-checker is harmful in specific cases: it decreases beliefs in true headlines that it mislabels as false and increases beliefs in false headlines that it is unsure about. On the positive side, AI fact-checking information increases the sharing intent for correctly labeled true headlines. When participants are given the option to view LLM fact checks and choose to do so, they are significantly more likely to share both true and false news but only more likely to believe false headlines. Our findings highlight an important source of potential harm stemming from AI applications and underscore the critical need for policies to prevent or mitigate such unintended consequences.

Human-Computer Interaction,Artificial Intelligence,Computers and Society
Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds

Joaquin Navajas,Tamara Niella,Gerry Garbulsky,Bahador Bahrami,Mariano Sigman

DOI: https://doi.org/10.1038/s41562-017-0273-4

IF: 24.252

2018-01-15

Nature Human Behaviour

Abstract:The aggregation of many independent estimates can outperform the most accurate individual judgement1–3. This centenarian finding1,2, popularly known as the 'wisdom of crowds'3, has been applied to problems ranging from the diagnosis of cancer4 to financial forecasting5. It is widely believed that social influence undermines collective wisdom by reducing the diversity of opinions within the crowd. Here, we show that if a large crowd is structured in small independent groups, deliberation and social influence within groups improve the crowd’s collective accuracy. We asked a live crowd (N = 5,180) to respond to general-knowledge questions (for example, "What is the height of the Eiffel Tower?"). Participants first answered individually, then deliberated and made consensus decisions in groups of five, and finally provided revised individual estimates. We found that averaging consensus decisions was substantially more accurate than aggregating the initial independent opinions. Remarkably, combining as few as four consensus choices outperformed the wisdom of thousands of individuals.

psychology, experimental,neurosciences,multidisciplinary sciences
Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking

Soham De,Michiel A. Bakker,Jay Baxter,Martin Saveski

2024-11-09

Abstract:X's Community Notes, a crowd-sourced fact-checking system, allows users to annotate potentially misleading posts. Notes rated as helpful by a diverse set of users are prominently displayed below the original post. While demonstrably effective at reducing misinformation's impact when notes are displayed, there is an opportunity for notes to appear on many more posts: for 91% of posts where at least one note is proposed, no notes ultimately achieve sufficient support from diverse users to be shown on the platform. This motivates the development of Supernotes: AI-generated notes that synthesize information from several existing community notes and are written to foster consensus among a diverse set of users. Our framework uses an LLM to generate many diverse Supernote candidates from existing proposed notes. These candidates are then evaluated by a novel scoring model, trained on millions of historical Community Notes ratings, selecting candidates that are most likely to be rated helpful by a diverse set of users. To test our framework, we ran a human subjects experiment in which we asked participants to compare the Supernotes generated by our framework to the best existing community notes for 100 sample posts. We found that participants rated the Supernotes as significantly more helpful, and when asked to choose between the two, preferred the Supernotes 75.2% of the time. Participants also rated the Supernotes more favorably than the best existing notes on quality, clarity, coverage, context, and argumentativeness. Finally, in a follow-up experiment, we asked participants to compare the Supernotes against LLM-generated summaries and found that the participants rated the Supernotes significantly more helpful, demonstrating that both the LLM-based candidate generation and the consensus-driven scoring play crucial roles in creating notes that effectively build consensus among diverse users.

Social and Information Networks
Automatic Fact-Checking Using Context and Discourse Information

Pepa Atanasova,Preslav Nakov,Lluís Màrquez,Alberto Barrón-Cedeño,Georgi Karadzhov,Tsvetomila Mihaylova,Mitra Mohtarami,James Glass

DOI: https://doi.org/10.1145/3297722

2019-08-04

Abstract:We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims, and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms of discourse cues and contextual features. For the check-worthiness estimation task, we focus on political debates, and we model the target claim in the context of the full intervention of a participant and the previous and the following turns in the debate, taking into account contextual meta information. For the fact-checking task, we focus on answer verification in a community forum, and we model the veracity of the answer with respect to the entire question--answer thread in which it occurs as well as with respect to other related posts from the entire forum. We develop annotated datasets for both tasks and we run extensive experimental evaluation, confirming that both types of information ---but especially contextual features--- play an important role.

Computation and Language,Artificial Intelligence
Fully Automated Fact Checking Using External Sources

Georgi Karadzhov,Preslav Nakov,Lluis Marquez,Alberto Barron-Cedeno,Ivan Koychev

DOI: https://doi.org/10.48550/arXiv.1710.00341

2017-10-01

Abstract:Given the constantly growing proliferation of false claims online in recent years, there has been also a growing research interest in automatically distinguishing false rumors from factually true claims. Here, we propose a general-purpose framework for fully-automatic fact checking using external sources, tapping the potential of the entire Web as a knowledge source to confirm or reject a claim. Our framework uses a deep neural network with LSTM text encoding to combine semantic kernels with task-specific embeddings that encode a claim together with pieces of potentially-relevant text fragments from the Web, taking the source reliability into account. The evaluation results show good performance on two different tasks and datasets: (i) rumor detection and (ii) fact checking of the answers to a question in community question answering forums.

Computation and Language
Community-based fact-checking reduces the spread of misleading posts on social media

Yuwei Chuai,Moritz Pilarski,Thomas Renault,David Restrepo-Amariles,Aurore Troussel-Clément,Gabriele Lenzini,Nicolas Pröllochs

2024-09-13

Abstract:Community-based fact-checking is a promising approach to verify social media content and correct misleading posts at scale. Yet, causal evidence regarding its effectiveness in reducing the spread of misinformation on social media is missing. Here, we performed a large-scale empirical study to analyze whether community notes reduce the spread of misleading posts on X. Using a Difference-in-Differences design and repost time series data for N=237,677 (community fact-checked) cascades that had been reposted more than 431 million times, we found that exposing users to community notes reduced the spread of misleading posts by, on average, 62.0%. Furthermore, community notes increased the odds that users delete their misleading posts by 103.4%. However, our findings also suggest that community notes might be too slow to intervene in the early (and most viral) stage of the diffusion. Our work offers important implications to enhance the effectiveness of community-based fact-checking approaches on social media.

Social and Information Networks
Does AI-Assisted Fact-Checking Disproportionately Benefit Majority Groups Online?

Terrence Neumann,Nicholas Wolczynski

DOI: https://doi.org/10.48550/arXiv.2302.03782

2023-02-10

Abstract:In recent years, algorithms have been incorporated into fact-checking pipelines. They are used not only to flag previously fact-checked misinformation, but also to provide suggestions about which trending claims should be prioritized for fact-checking - a paradigm called `check-worthiness.' While several studies have examined the accuracy of these algorithms, none have investigated how the benefits from these algorithms (via reduction in exposure to misinformation) are distributed amongst various online communities. In this paper, we investigate how diverse representation across multiple stages of the AI development pipeline affects the distribution of benefits from AI-assisted fact-checking for different online communities. We simulate information propagation through the network using our novel Topic-Aware, Community-Impacted Twitter (TACIT) simulator on a large Twitter followers network, tuned to produce realistic cascades of true and false information across multiple topics. Finally, using simulated data as a test bed, we implement numerous algorithmic fact-checking interventions that explicitly account for notions of diversity. We find that both representative and egalitarian methods for sampling and labeling check-worthiness model training data can lead to network-wide benefit concentrated in majority communities, while incorporating diversity into how fact-checkers use algorithmic recommendations can actively reduce inequalities in benefits between majority and minority communities. These findings contribute to an important conversation around the responsible implementation of AI-assisted fact-checking by social media platforms and fact-checking organizations.

Computers and Society,Social and Information Networks
Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?

Veronica Chatrath,Marcelo Lotif,Shaina Raza

2024-11-09

Abstract:Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.

Computation and Language,Artificial Intelligence
The perils and promises of fact-checking with large language models

Dorian Quelle,Alexandre Bovet

DOI: https://doi.org/10.3389/frai.2024.1341697

2024-02-07

Abstract:Automated fact-checking, using machine learning to verify claims, has grown vital as misinformation spreads beyond human fact-checking capacity. Large language models (LLMs) like GPT-4 are increasingly trusted to write academic papers, lawsuits, and news articles and to verify information, emphasizing their role in discerning truth from falsehood and the importance of being able to verify their outputs. Understanding the capacities and limitations of LLMs in fact-checking tasks is therefore essential for ensuring the health of our information ecosystem. Here, we evaluate the use of LLM agents in fact-checking by having them phrase queries, retrieve contextual data, and make decisions. Importantly, in our framework, agents explain their reasoning and cite the relevant sources from the retrieved context. Our results show the enhanced prowess of LLMs when equipped with contextual information. GPT-4 outperforms GPT-3, but accuracy varies based on query language and claim veracity. While LLMs show promise in fact-checking, caution is essential due to inconsistent accuracy. Our investigation calls for further research, fostering a deeper comprehension of when agents succeed and when they fail.
Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

Md Momen Bhuiyan,Amy X. Zhang,Connie Moon Sehat,Tanushree Mitra

DOI: https://doi.org/10.1145/3415164

2020-08-21

Abstract:Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.

Human-Computer Interaction

Scaling up fact-checking using the wisdom of crowds

Fact-Checking at Scale with DimensionRank

The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale

Searching For or Reviewing Evidence Improves Crowdworkers' Misinformation Judgments and Reduces Partisan Bias

Who Checks the Checkers? Exploring Source Credibility in Twitter's Community Notes

Check-COVID: Fact-Checking COVID-19 News Claims with Scientific Evidence

Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

Some Observations on Fact-Checking Work with Implications for Computational Support

Checking the Fact-Checkers: the Role of Source Type, Perceived Credibility, and Individual Differences in Fact-Checking Effectiveness

HybridFC: A Hybrid Fact-Checking Approach for Knowledge Graphs

The State of Human-centered NLP Technology for Fact-checking

Fact-checking information from large language models can decrease headline discernment

Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds

Supernotes: Driving Consensus in Crowd-Sourced Fact-Checking

Automatic Fact-Checking Using Context and Discourse Information

Fully Automated Fact Checking Using External Sources

Community-based fact-checking reduces the spread of misleading posts on social media

Does AI-Assisted Fact-Checking Disproportionately Benefit Majority Groups Online?

Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?

The perils and promises of fact-checking with large language models

Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria