Abstract:Reinforcement Learning views the maximization of rewards and avoidance of punishments as central to explaining goal-directed behavior. However, over a life, organisms will need to learn about many different aspects of the world's structure: the states of the world and state-vector transition dynamics. The number of combinations of states grows exponentially as an agent incorporates new knowledge, and there is no obvious weighted combination of pre-existing rewards or costs defined for a given combination of states, as such a weighting would need to encode information about good and bad combinations prior to an agent's experience in the world. Therefore, we must develop more naturalistic accounts of behavior and motivation in large state-spaces. We show that it is possible to use only the intrinsic motivation metric of empowerment, which measures the agent's capacity to realize many possible futures under a transition operator. We propose to scale empowerment to hierarchical state-spaces by using Operator Bellman Equations. These equations produce state-time feasibility functions, which are compositional hierarchical state-time transition operators that map an initial state and time when an agent begins a policy to the final states and times of completing a goal. Because these functions are hierarchical operators we can define hierarchical empowerment measures on them. An agent can then optimize plans to distant states and times to maximize its hierarchical empowerment-gain, allowing it to discover goals that bring about a more favorable coupling of its internal structure (physiological states) to its external environment (world structure & spatial state). Life-long agents could therefore be primarily animated by principles of compositionality and empowerment, exhibiting self-concern for the growth & maintenance of their own structural integrity without recourse to reward-maximization.

Simplifying social learning

Social Concepts Simplify Complex Reinforcement Learning

Humans flexibly integrate social information despite interindividual differences in reward

Social learning spontaneously emerges by searching optimal heuristics with deep reinforcement learning

Why does individual learning endure when crowds are wiser?

A Minimal Model of the Interaction of Social and Individual learning

Flexible social inference facilitates targeted social learning when rewards are not observable

Emergent Social Learning via Multi-agent Reinforcement Learning

Shedding light on social learning

Behavioral and Neural Properties of Social Reinforcement Learning.

Meta-control of social learning strategies

Neural Mechanisms of Social Learning and Decision-Making

Intelligent Social Learning

Social Learning in a Human Society: An Experimental Study

Learning and choosing in an uncertain world: An investigation of the explore-exploit dilemma in static and dynamic environments

Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning

Reinforcement learning facilitates an optimal interaction intensity for cooperation

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Understanding the World to Solve Social Dilemmas Using Multi-Agent Reinforcement Learning

Modeling the formation of social conventions from embodied real-time interactions

Unlearnable Games and "Satisficing'' Decisions: A Simple Model for a Complex World