Abstract:Interactions between an intelligent software agent (ISA) and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user’s access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality (and deploying concepts from artificial intelligence, behavioural economics, control theory, and game theory), we frame these interactions as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user’s behaviour towards outcomes that maximise the ISA’s utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction (i.e. deception, coercion, trading, and nudging), as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings.

Learning with Digital Agents: An Analysis based on the Activity Theory

Learning through Dialogue Interactions by Asking Questions

Animated Pedagogical Agents Enhance Learning Outcomes and Brain Activity During Learning.

Pedagogical Agents: Back to the Future

Moving forward with activity theory in a digital world

The role and construction of educational agents in distance learning environments

EduAgent: Generative Student Agents in Learning

Social Learning through Interactions with Other Agents: A Survey

ActiveAI: Introducing AI Literacy for Middle School Learners with Goal-based Scenario Learning

Exploring Social Learning in Collaborative Augmented Reality With Pedagogical Agents as Learning Companions

Natural Language Communication with a Teachable Agent

A Digital Game based Learning Approach for Effective Curriculum Transaction for Teaching-Learning of Artificial Intelligence and Machine Learning

An Analysis of the Interaction Between Intelligent Software Agents and Human Users

Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions

A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

Intelligent Agents: A Physics Education Opportunity in Latin-America

Educational Psychology Aspects of Learning with Chatbots without Artificial Intelligence: Suggestions for Designers

Learner Perceptions of Artificial Intelligence-Generated Pedagogical Agents in Language Learning Videos: Embodiment Effects on Technology Acceptance

Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Generative Artificial Intelligence in Learning Analytics: Contextualising Opportunities and Challenges through the Learning Analytics Cycle