From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agents

Mohit Chandra,Suchismita Naik,Denae Ford,Ebele Okoli,Munmun De Choudhury,Mahsa Ershadi,Gonzalo Ramos,Javier Hernandez,Ananya Bhattacharjee,Shahed Warreth,Jina Suh
2024-12-11
Abstract:Recent gain in popularity of AI conversational agents has led to their increased use for improving productivity and supporting well-being. While previous research has aimed to understand the risks associated with interactions with AI conversational agents, these studies often fall short in capturing the lived experiences. Additionally, psychological risks have often been presented as a sub-category within broader AI-related risks in past taxonomy works, leading to under-representation of the impact of psychological risks of AI use. To address these challenges, our work presents a novel risk taxonomy focusing on psychological risks of using AI gathered through lived experience of individuals. We employed a mixed-method approach, involving a comprehensive survey with 283 individuals with lived mental health experience and workshops involving lived experience experts to develop a psychological risk taxonomy. Our taxonomy features 19 AI behaviors, 21 negative psychological impacts, and 15 contexts related to individuals. Additionally, we propose a novel multi-path vignette based framework for understanding the complex interplay between AI behaviors, psychological impacts, and individual user contexts. Finally, based on the feedback obtained from the workshop sessions, we present design recommendations for developing safer and more robust AI agents. Our work offers an in-depth understanding of the psychological risks associated with AI conversational agents and provides actionable recommendations for policymakers, researchers, and developers.
Human-Computer Interaction,Artificial Intelligence,Computers and Society
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to address the psychological risks associated with the use of AI - powered conversational agents. Specifically, existing research has the following shortcomings in understanding these psychological risks: 1. **Lack of Capturing of Real - Life Experiences**: Previous studies often failed to fully capture users' real - life experiences (lived experiences), resulting in an incomplete understanding of psychological risks. 2. **Underestimation of Psychological Risks**: Psychological risks are usually discussed as a sub - category of broader AI - related risks, thus ignoring their uniqueness and importance. 3. **Neglect of Situational Complexity**: The impact of psychological risks is influenced by multiple situational factors, which are often overlooked in existing research. To fill these gaps, this paper proposes a taxonomy of psychological risks based on real - life experiences by combining quantitative and qualitative methods, and develops a multi - path vignette framework to better understand and address the psychological risks brought by AI - powered conversational agents. In addition, the authors also provide design recommendations based on the research results to help developers and policymakers create safer and more robust AI systems. ### Main Contributions of the Research 1. **Proposing a New Taxonomy of Psychological Risks**: This taxonomy includes three main components - AI behavior, negative psychological impacts, and situational factors, and it details their interactions. 2. **Developing a Multi - Path Vignette Framework**: This framework shows the complex relationships between different AI behaviors, impact categories, and situations, providing an intuitive tool for understanding these risks. 3. **Providing Practical Design Recommendations**: Based on workshop feedback, the authors provide practical design recommendations for AI practitioners and developers to design safer AI - powered conversational agents. ### Research Methods - **Phase 1: Development of the Psychological Risk Taxonomy** - **Survey Research**: Through a survey involving 283 individuals with mental health experiences, their negative psychological experiences during interactions with AI - powered conversational agents were collected. - **Workshop Validation**: Through three workshops with 7 highly experienced experts, the taxonomy was verified and refined. - **Phase 2: Development of Future AI Design Recommendations** - **Workshop Discussions**: Based on the results of phase 1, further discussions were held on how to design safer AI systems, and specific design recommendations were proposed. Through this method, the authors not only revealed the psychological risks that AI - powered conversational agents may bring, but also provided valuable guidance for future AI design.