Abstract:The problem of sparse rewards is one of the hardest challenges in contemporary reinforcement learning. Hierarchical reinforcement learning (HRL) tackles this problem by using a set of temporally-extended actions, or options, each of which has its own subgoal. These subgoals are normally handcrafted for specific tasks. Here, though, we introduce a generic class of subgoals with broad applicability in the visual domain. Underlying our approach (in common with work using "auxiliary tasks") is the hypothesis that the ability to control aspects of the environment is an inherently useful skill to have. We incorporate such subgoals in an end-to-end hierarchical reinforcement learning system and test two variants of our algorithm on a number of games from the Atari suite. We highlight the advantage of our approach in one of the hardest games -- Montezuma's revenge -- for which the ability to handle sparse rewards is key. Our agent learns several times faster than the current state-of-the-art HRL agent in this game, reaching a similar level of performance. UPDATE 22/11/17: We found that a standard A3C agent with a simple shaped reward, i.e. extrinsic reward + feature control intrinsic reward, has comparable performance to our agent in Montezuma Revenge. In light of the new experiments performed, the advantage of our HRL approach can be attributed more to its ability to learn useful features from intrinsic rewards rather than its ability to explore and reuse abstracted skills with hierarchical components. This has led us to a new conclusion about the result.

Successor Feature Neural Episodic Control

Temporally extended successor feature neural episodic control

Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning

Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

Continual Reinforcement Learning with Multi-Timescale Successor Features

Deep Successor Feature Learning for Text Generation.

Combining Behaviors with the Successor Features Keyboard

Successor-Predecessor Intrinsic Exploration

Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer

Learning Successor Features the Simple Way

Multi-Task Reinforcement Learning in Continuous Control with Successor Feature-Based Concurrent Composition

Efficient Reinforcement-Learning Control Algorithm Using Experience Reuse

Neural Prioritisation of Past Solutions Supports Generalisation

Efficient Deep Reinforcement Learning Via Policy-Extended Successor Feature Approximator

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

Generative Adversarial Implicit Successor Representation

Continuous Episodic Control

Efficient Exploration for Multi-Agent Reinforcement Learning Via Transferable Successor Features

Never Give Up: Learning Directed Exploration Strategies

Autonomous learning of features for control: Experiments with embodied and situated agents