Large Language Models Can Argue in Convincing Ways About Politics, But Humans Dislike AI Authors: implications for Governance

Alexis Palmer,Arthur Spirling,Alexis PalmerArthur Spirlinga Department of Politics,New York University,New York,NY,USb Department of Politics,Princeton University,Princeton,NJ,USAlexis Palmer is a PhD candidate at New York University.Arthur Spirling is the Class of 1987 Professor of Politics at Princeton University.
DOI: https://doi.org/10.1080/00323187.2024.2335471
2024-04-18
Political Science
Abstract:All politics relies on rhetorical appeals, and the ability to make arguments is considered perhaps uniquely human. But as recent times have seen successful large language model (LLM) applications to similar endeavours, we explore whether these approaches can out-compete humans in making appeals for/against various positions in US politics. We curate responses from crowdsourced workers and an LLM and place them in competition with one another. Human (crowd) judges make decisions about the relative strength of their (human v machine) efforts. We have several empirical 'possibility' results. First, LLMs can produce novel arguments that convince independent judges at least on a par with human efforts. Yet when informed about an orator's true identity, judges show a preference for human over LLM arguments. This may suggest voters view such models as potentially dangerous; we think politicians should be aware of related 'liar's dividend' concerns.
political science
What problem does this paper attempt to address?