User experience with conversational agent: a systematic review of assessment methods

Carla Tubin,João Pedro Mazuco Rodriguez,Ana Carolina Bertoletti de Marchi
DOI: https://doi.org/10.1080/0144929X.2021.2001047
2021-12-14
Behaviour and Information Technology
Abstract:Conversational agents are becoming popular for providing a more natural and realistic user experience. New studies have become significant to understand how to assess this experience, mainly because of the increase in applications of this nature. We systematically reviewed the literature to identify how the user experience is assessed when interacting with conversational agents. A total of 443 studies were identified in the ACM, IEEE, Springer, and Scopus databases. Of these, 27 studies met the eligibility criteria. Most studies used their own evaluation methods, without adopting questionnaires validated for UX evaluation. Few studies used assessment tools before participants interacted with agents, and only two carried out assessments before, during, and after use. The results of the assessments can be better if specific instruments for UX are adopted. Furthermore, it is necessary to assess the experience at different times and use combined methods, to understand aspects related to the participants' feelings and behaviours.
ergonomics,computer science, cybernetics
What problem does this paper attempt to address?