Abstract:To achieve the ambitious goals of artificial intelligence, reinforcement learning must include planning with a model of the world that is abstract in state and time. Deep learning has made progress with state abstraction, but temporal abstraction has rarely been used, despite extensively developed theory based on the options framework. One reason for this is that the space of possible options is immense, and the methods previously proposed for option discovery do not take into account how the option models will be used in planning. Options are typically discovered by posing subsidiary tasks, such as reaching a bottleneck state or maximizing the cumulative sum of a sensory signal other than reward. Each subtask is solved to produce an option, and then a model of the option is learned and made available to the planning process. In most previous work, the subtasks ignore the reward on the original problem, whereas we propose subtasks that use the original reward plus a bonus based on a feature of the state at the time the option terminates. We show that option models obtained from such reward-respecting subtasks are much more likely to be useful in planning than eigenoptions, shortest path options based on bottleneck states, or reward-respecting options generated by the option-critic. Reward respecting subtasks strongly constrain the space of options and thereby also provide a partial solution to the problem of option discovery. Finally, we show how values, policies, options, and models can all be learned online and off-policy using standard algorithms and general value functions.

Intrinsically motivated option learning: a comparative study of recent methods

Salience Interest Option: Temporal abstraction with salience interest functions

Reward-Respecting Subtasks for Model-Based Reinforcement Learning

LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework

An agent with a sense of direction for option discovery in hierarchical reinforcement learning

An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey

Intrinsically-Motivated Reinforcement Learning: A Brief Introduction

An Autonomous Non-monolithic Agent with Multi-mode Exploration based on Options Framework

Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments

Option Compatible Reward Inverse Reinforcement Learning

Variational Intrinsic Control

A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning

Empowering the Diversity and Individuality of Option: Residual Soft Option Critic Framework

An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments

LISPR: An Options Framework for Policy Reuse with Reinforcement Learning

Autonomous Discovery and Creation of Options in Hierarchical Reinforcement Learning

A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment

Applying Reinforcement Learning to Option Pricing and Hedging

Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates

When Waiting is not an Option : Learning Options with a Deliberation Cost

Show me the Way: Intrinsic Motivation from Demonstrations