I Am Not Your Typical Chatbot: Hedonic and Utilitarian Evaluation of Open-Domain Chatbots
Joo-Wha Hong Katrin Fischer Donggyu Kim Justin Hyundong Cho Yuan Sun a Department of Applied Artificial Intelligence,Sungkyunkwan University,Seoul,South Koreab Annenberg School for Communication and Journalism,University of Southern California,Los Angeles,CA,USAc Information Sciences Institute,University of Southern California,Los Angeles,CA,USAd College of Journalism,Communications University of Florida,Gainesville,FL,USAJoo-Wha Hong is an assistant professor at Sungkyunkwan University. His research primarily focuses on the social influence of autonomous machines,particularly artificial intelligence (AI),with an emphasis on the cognitive and psychological aspects of human-AI interaction. He explores how individuals perceive machines in social roles and develop relationships with them.Katrin Fischer is a quantitative social scientist and human-machine communication scholar with a focus on human-robot interaction. Her research explores the cognitive,social,and technology-based processes that affect expectations and decision-making when humans communicate with (embodied) artificial intelligence systems.Donggyu Kim is a PhD student at the Annenberg School for Communication and Journalism at the University of Southern California. He studies the psychological underpinnings of human-AI interaction. His recent research focuses on how AI-mediated communication influences message persuasion and how Generative AI transforms work processes.Justin Hyundong Cho is a PhD candidate at University of Southern California's Information Sciences Institute. His research is centered around improving and understanding the role of language models in human-AI interactions to develop accessible and reliable digital assistants.Yuan Sun is an Assistant Professor of Advertising at the University of Florida. Her research focuses on the role of artificial intelligence in strategic communication,exploring algorithmic transparency,user trust in AI media interfaces,and consumer interactions with AI in digital health,advertising,and news consumption.
DOI: https://doi.org/10.1080/10447318.2024.2416016
IF: 4.92
2024-10-23
International Journal of Human-Computer Interaction
Abstract:With the development of natural language processing (NLP), open-domain chatbots can operate as companions. The success of open-domain chatbots depends on understanding how positive and negative expectancy violations affect both utilitarian and hedonic gratification, which in turn influences user satisfaction and continued use. Therefore, this study examines how violations of expectations influence the hedonic and utilitarian evaluations of open-domain chatbots using the expectancy-violation theory. Participants ( n = 204) interacted with an improvising AI chatbot and reported their satisfaction. Results indicated that both hedonic and utilitarian satisfaction were higher when negative expectations were met compared to when expectations were violated. However, no significant difference was found in hedonic gratification between the chatbot's expected performance and positively unexpected performance. Conversations meeting expectations elicited more utilitarian satisfaction than positively surprising interactions. These findings highlight the dynamics between value types and expectation violations in AI chatbot evaluations.
computer science, cybernetics,ergonomics