Smooth Talking: Generative Versus Scripted Coaching Chatbot Adoption and Efficacy Comparison

Nicky H. D. Terblanche
DOI: https://doi.org/10.1080/10447318.2024.2423125
IF: 4.92
2024-11-13
International Journal of Human-Computer Interaction
Abstract:Workplace coaching is an essential individual and organizational development intervention, but high costs limit its application. AI coaching chatbots could scale coaching, however, traditional scripted chatbots lacks mainstream adoption due to limited context awareness and robotic responses. Recent advances in large language models and generative AI offer an alternative, however, generative AI is linked to risks and controversies. Should coaching chatbots use generative AI? We compared a scripted, rules-based coaching chatbot (ScriptBot, n = 126) with a generative AI chatbot (GenBot, n = 116) in a randomized controlled trail experiment. We measured technology adoption (UTAUT2), self-determination (autonomy, competence, relatedness) and goal attainment. For all technology adoption and self-determination variables, GenBot outperformed ScriptBot. However, for goal attainment there was no significant difference. Using generative AI in coaching chatbots improves adoption and coaching outcomes, but for highly structured interactions such as goal attainment, scripted conversations (with lower risk) seem to work just as well.
computer science, cybernetics,ergonomics
What problem does this paper attempt to address?