Abstract:Reinforcement Learning views the maximization of rewards and avoidance of punishments as central to explaining goal-directed behavior. However, over a life, organisms will need to learn about many different aspects of the world's structure: the states of the world and state-vector transition dynamics. The number of combinations of states grows exponentially as an agent incorporates new knowledge, and there is no obvious weighted combination of pre-existing rewards or costs defined for a given combination of states, as such a weighting would need to encode information about good and bad combinations prior to an agent's experience in the world. Therefore, we must develop more naturalistic accounts of behavior and motivation in large state-spaces. We show that it is possible to use only the intrinsic motivation metric of empowerment, which measures the agent's capacity to realize many possible futures under a transition operator. We propose to scale empowerment to hierarchical state-spaces by using Operator Bellman Equations. These equations produce state-time feasibility functions, which are compositional hierarchical state-time transition operators that map an initial state and time when an agent begins a policy to the final states and times of completing a goal. Because these functions are hierarchical operators we can define hierarchical empowerment measures on them. An agent can then optimize plans to distant states and times to maximize its hierarchical empowerment-gain, allowing it to discover goals that bring about a more favorable coupling of its internal structure (physiological states) to its external environment (world structure & spatial state). Life-long agents could therefore be primarily animated by principles of compositionality and empowerment, exhibiting self-concern for the growth & maintenance of their own structural integrity without recourse to reward-maximization.

Learning telic-controllable state representations

States as goal-directed concepts: an epistemic approach to state-representation learning

Towards Learning Controllable Representations of Physical Systems

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

Unsupervised Control Through Non-Parametric Discriminative Rewards

Learning with Language-Guided State Abstractions

Taming AI Bots: Controllability of Neural States in Large Language Models

Learning Optimal Strategies for Temporal Tasks in Stochastic Games

Learning Action-based Representations Using Invariance

An Overview of Natural Language State Representation for Reinforcement Learning

Interactive Learning of State Representation through Natural Language Instruction and Explanation

Balancing control: a Bayesian interpretation of habitual and goal-directed behavior

Inverse Reinforcement Learning: A Control Lyapunov Approach

Effective Representation Learning is More Effective in Reinforcement Learning Than You Think

Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning

Bridging State and History Representations: Understanding Self-Predictive RL

Information is Power: Intrinsic Control via Information Capture

Formalising the intentional stance: attributing goals and beliefs to stochastic processes

Model-based Reinforcement Learning from Signal Temporal Logic Specifications

Dream to Control: Learning Behaviors by Latent Imagination

Task-Induced Representation Learning