Abstract:The rapid advancement of artificial intelligence (AI) systems suggests that artificial general intelligence (AGI) systems may soon arrive. Many researchers are concerned that AIs and AGIs will harm humans via intentional misuse (AI-misuse) or through accidents (AI-accidents). In respect of AI-accidents, there is an increasing effort focused on developing algorithms and paradigms that ensure AI systems are aligned to what humans intend, e.g. AI systems that yield actions or recommendations that humans might judge as consistent with their intentions and goals. Here we argue that alignment to human intent is insufficient for safe AI systems and that preservation of long-term agency of humans may be a more robust standard, and one that needs to be separated explicitly and a priori during optimization. We argue that AI systems can reshape human intention and discuss the lack of biological and psychological mechanisms that protect humans from loss of agency. We provide the first formal definition of agency-preserving AI-human interactions which focuses on forward-looking agency evaluations and argue that AI systems - not humans - must be increasingly tasked with making these evaluations. We show how agency loss can occur in simple environments containing embedded agents that use temporal-difference learning to make action recommendations. Finally, we propose a new area of research called "agency foundations" and pose four initial topics designed to improve our understanding of agency in AI-human interactions: benevolent game theory, algorithmic foundations of human rights, mechanistic interpretability of agency representation in neural-networks and reinforcement learning from internal states.

On the Ethics of Building AI in a Responsible Manner

Current cases of AI misalignment and their implications for future risks

Artificial Intelligence, Values and Alignment

Responsible (use of) AI

The Alignment Problem from a Deep Learning Perspective

There and Back Again: The AI Alignment Paradox

Safety without alignment

Risk Alignment in Agentic AI Systems

Adaptive AI Alignment: Established Resources for Aligning Machine Learning with Human Intentions and Values in Changing Environments

Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

Responsible AI by Design in Practice

AI Value Alignment Problem: The Clear and Present Danger

The obscure politics of artificial intelligence: a Marxian socio-technical critique of the AI alignment problem thesis

AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

Filling gaps in trustworthy development of AI

The Alignment Problem in Context

AI Alignment and Social Choice: Fundamental Limitations and Policy Implications

The Ethics of AI: Navigating the Moral Dilemmas of Artificial Intelligence

Implementing Responsible AI: Tensions and Trade-Offs Between Ethics Aspects

AI Alignment with Changing and Influenceable Reward Functions

On the Undecidability of Artificial Intelligence Alignment: Machines that Halt