PERSONALIZATION: Why the relational modes between Generative AI chatbots and human users are critical factors for product design and safety

Duane Valz

DOI: https://doi.org/10.2139/ssrn.4468899

2023-01-01

SSRN Electronic Journal

Abstract:Our experience of Generative AI platforms is through applications such as ChatGPT, a general purpose chatbot that can capably respond to a broad diversity of natural language prompts. Generative AI chatbots are designed to be conversational applications. As such, their ability to keep us engaged depends in part on how suitably personalized their outputs are to both our prompts and our underlying motivations. Correspondingly, reported instances of adverse user experiences with chatbots have involved outputs that are personalized in coercive or disturbing ways. As a design focus for Generative AI chatbots, personalization also brings with it psychological safety risks. Research has long shown that humans are highly susceptible to anthropomorphizing computation systems with conversational capabilities, an important facet of understanding why chatbots can both satisfy and threaten a user’s sense of well-being. Recent studies reveal that Generative AI Chatbots can be highly proficient at understanding and appealing to distinct human personality traits. Nonetheless, such chatbots are trained on corpora that include toxic, abusive, and manipulative language, and this is the primary source of psychological safety risk for users predisposed to anthropomorphizing conversational computation systems. Given the current AI industry trajectory to deepen user engagement with chatbots, it is important to consider both the promises and perils of personalization. In this regard, I present a categorization of Generative AI chatbots based on the extent to which their design and use is premised on personalized user experiences, with examples of in-market chatbots that fall into each category. I then present a rubric for thinking about the psychological safety risks that may be posed by a Generative AI chatbot, considering the extent to which the chatbot is designed to provide, or its users intend to experience, personalized interactions. I conclude by examining the possibilities for deepening personalization proficiency of chatbots with different types of user data and the issues these raise for further research, legal compliance, industry best practices development, and public policy development.