Abstract:The problem of sparse rewards is one of the hardest challenges in contemporary reinforcement learning. Hierarchical reinforcement learning (HRL) tackles this problem by using a set of temporally-extended actions, or options, each of which has its own subgoal. These subgoals are normally handcrafted for specific tasks. Here, though, we introduce a generic class of subgoals with broad applicability in the visual domain. Underlying our approach (in common with work using "auxiliary tasks") is the hypothesis that the ability to control aspects of the environment is an inherently useful skill to have. We incorporate such subgoals in an end-to-end hierarchical reinforcement learning system and test two variants of our algorithm on a number of games from the Atari suite. We highlight the advantage of our approach in one of the hardest games -- Montezuma's revenge -- for which the ability to handle sparse rewards is key. Our agent learns several times faster than the current state-of-the-art HRL agent in this game, reaching a similar level of performance. UPDATE 22/11/17: We found that a standard A3C agent with a simple shaped reward, i.e. extrinsic reward + feature control intrinsic reward, has comparable performance to our agent in Montezuma Revenge. In light of the new experiments performed, the advantage of our HRL approach can be attributed more to its ability to learn useful features from intrinsic rewards rather than its ability to explore and reuse abstracted skills with hierarchical components. This has led us to a new conclusion about the result.

Deep Reinforcement Learning from Hierarchical Preference Design

Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?

Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

Reinforcement Learning from Diverse Human Preferences

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

Deep Reinforcement Learning with Hierarchical Structures.

Hierarchical Reinforcement Learning with Attention Reward

Temporal-adaptive Hierarchical Reinforcement Learning

Advances in Hierarchical Reinforcement Learning

Tiered Reward: Designing Rewards for Specification and Fast Learning of Desired Behavior

Summarize of hierarchical reinforcement learning

Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

Hierarchical Reinforcement Learning: A Survey and Open Research Challenges

A Reward Optimization Method Based on Action Subrewards in Hierarchical Reinforcement Learning

Optimal Hierarchical Learning Path Design with Reinforcement Learning

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Feature Control as Intrinsic Motivation for Hierarchical Reinforcement Learning

PROVABLY BENEFITS OF DEEP HIERARCHICAL RL

Boosting Reinforcement Learning via Hierarchical Game Playing With State Relay

Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning