Abstract:This article presents a comparative analysis of the ability of two large language model (LLM)-based chatbots, ChatGPT and Bing Chat, recently rebranded to Microsoft Copilot, to detect veracity of political information. We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ+ related debates. We compare how the chatbots perform in high- and low-resource languages by using prompts in English, Russian, and Ukrainian. Furthermore, we explore the ability of chatbots to evaluate statements according to political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts. We also systematically test how such evaluations are influenced by source bias which we model by attributing specific claims to various political and social actors. The results show high performance of ChatGPT for the baseline veracity evaluation task, with 72 percent of the cases evaluated correctly on average across languages without pre-training. Bing Chat performed worse with a 67 percent accuracy. We observe significant disparities in how chatbots evaluate prompts in high- and low-resource languages and how they adapt their evaluations to political communication concepts with ChatGPT providing more nuanced outputs than Bing Chat. Finally, we find that for some veracity detection-related tasks, the performance of chatbots varied depending on the topic of the statement or the source to which it is attributed. These findings highlight the potential of LLM-based chatbots in tackling different forms of false information in online environments, but also points to the substantial variation in terms of how such potential is realized due to specific factors, such as language of the prompt or the topic.

LLMs as information warriors? Auditing how LLM-powered chatbots tackle disinformation about Russia's war in Ukraine

In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?

Combating misinformation in the age of LLMs: Opportunities and challenges

Lying Blindly: Bypassing ChatGPT's Safeguards to Generate Hard-to-Detect Disinformation Claims at Scale

Can LLM-Generated Misinformation Be Detected?

LLMs Among Us: Generative AI Participating in Digital Discourse

Disinformation Capabilities of Large Language Models

LLM Echo Chamber: personalized and automated disinformation

Human Perception of LLM-generated Text Content in Social Media Environments

Misinforming LLMs: vulnerabilities, challenges and opportunities

LLM generated responses to mitigate the impact of hate speech

Large language models can consistently generate high-quality content for election disinformation operations

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

The Dark Side of Language Models: Exploring the Potential of LLMs in Multimedia Disinformation Generation and Dissemination

Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts

Large Language Models as Instruments of Power: New Regimes of Autonomous Manipulation and Control

Understanding Knowledge Drift in LLMs through Misinformation

When LLMs Go Online: The Emerging Threat of Web-Enabled LLMs

LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users

Large Language Models Reveal Information Operation Goals, Tactics, and Narrative Frames