Abstract:Reinforcement Learning views the maximization of rewards and avoidance of punishments as central to explaining goal-directed behavior. However, over a life, organisms will need to learn about many different aspects of the world's structure: the states of the world and state-vector transition dynamics. The number of combinations of states grows exponentially as an agent incorporates new knowledge, and there is no obvious weighted combination of pre-existing rewards or costs defined for a given combination of states, as such a weighting would need to encode information about good and bad combinations prior to an agent's experience in the world. Therefore, we must develop more naturalistic accounts of behavior and motivation in large state-spaces. We show that it is possible to use only the intrinsic motivation metric of empowerment, which measures the agent's capacity to realize many possible futures under a transition operator. We propose to scale empowerment to hierarchical state-spaces by using Operator Bellman Equations. These equations produce state-time feasibility functions, which are compositional hierarchical state-time transition operators that map an initial state and time when an agent begins a policy to the final states and times of completing a goal. Because these functions are hierarchical operators we can define hierarchical empowerment measures on them. An agent can then optimize plans to distant states and times to maximize its hierarchical empowerment-gain, allowing it to discover goals that bring about a more favorable coupling of its internal structure (physiological states) to its external environment (world structure & spatial state). Life-long agents could therefore be primarily animated by principles of compositionality and empowerment, exhibiting self-concern for the growth & maintenance of their own structural integrity without recourse to reward-maximization.

Reward is not enough: can we liberate AI from the reinforcement learning paradigm?

Three Dogmas of Reinforcement Learning

Do Artificial Reinforcement-Learning Agents Matter Morally?

Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning

When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback

Giving Up Control: Neurons as Reinforcement Learning Agents

Learning Purposeful Behaviour in the Absence of Rewards

A General, Evolution-Inspired Reward Function for Social Robotics

Knowledge is reward: Learning optimal exploration by predictive reward cashing

Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback

Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning

Human irrationality: both bad and good for reward inference

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Using deep reinforcement learning to promote sustainable human behaviour on a common pool resource problem

The History and Risks of Reinforcement Learning and Human Feedback

Reinforcement Learning in Practice: Opportunities and Challenges

Axioms for AI Alignment from Human Feedback

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Breaching the Bottleneck: Evolutionary Transition from Reward-Driven Learning to Reward-Agnostic Domain-Adapted Learning in Neuromodulated Neural Nets

Grounding Artificial Intelligence in the Origins of Human Behavior

Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning