Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content

Martin Huschens,Martin Briesch,Dominik Sobania,Franz Rothlauf

DOI: https://doi.org/10.48550/arXiv.2309.02524

2023-09-06

Abstract:This paper examines how individuals perceive the credibility of content originating from human authors versus content generated by large language models, like the GPT language model family that powers ChatGPT, in different user interface versions. Surprisingly, our results demonstrate that regardless of the user interface presentation, participants tend to attribute similar levels of credibility. While participants also do not report any different perceptions of competence and trustworthiness between human and AI-generated content, they rate AI-generated content as being clearer and more engaging. The findings from this study serve as a call for a more discerning approach to evaluating information sources, encouraging users to exercise caution and critical thinking when engaging with content generated by AI systems.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to explore the differences in individuals' credibility perception of content generated by human authors and that generated by large - language models (such as ChatGPT driven by the GPT language model family) in different user - interface versions. Specifically, through an online survey, the researchers analyzed the participants' credibility perception of text excerpts, which were presented in different user - interface settings (ChatGPT interface, plain - text interface, Wikipedia interface), while manipulating the source of the text: human - generated or generated by large - language models ("LLM - generated").

The main findings of the study are as follows:

- Regardless of how the user interface presents, participants tend to assign similar levels of credibility to the content.
- Participants did not report any significant differences in perceived ability and credibility between human - generated content and AI - generated content.
- However, participants considered AI - generated content to be clearer and more appealing.

These findings highlight an urgent problem: although applications based on large - language models have made remarkable progress in content generation, there are still potential risks of generating errors, misunderstandings, and even hallucinatory behaviors in these applications. Therefore, the public is increasingly relying on these platforms for information, which brings significant risks. The study shows that users do not seem to distinguish the source of the content, which may lead to the spread of false information and the erosion of critical - thinking skills, especially when considering the long - term impacts.

In conclusion, this study calls for a more cautious attitude towards evaluating information sources and encourages users to exercise caution and critical thinking when exposed to content generated by AI systems.