Abstract:Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. Various techniques have been proposed to improve performance in goal-conditioned environments, such as automatic curriculum generation and goal relabeling. In this work, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. In particular: the current Q-value function and the target Q-value estimate are both functions of the goal, and we would like to train the Q-value function to match its target for all goals. We therefore apply Gradient-Based Attention Transfer (Zagoruyko and Komodakis 2017), a knowledge distillation technique, to the Q-function update. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals, where the agent can attain a reward by achieving any one of a large set of objectives, all specified at test time. Finally, to provide theoretical support, we give examples of classes of environments where (under some assumptions) standard off-policy algorithms such as DDPG require at least O(d^2) replay buffer transitions to learn an optimal policy, while our proposed technique requires only O(d) transitions, where d is the dimensionality of the goal and state space. Code is available at <a class="link-external link-https" href="https://github.com/alevine0/ReenGAGE" rel="external noopener nofollow">this https URL</a>.

Many-Goals Reinforcement Learning

Generating Attentive Goals for Prioritized Hindsight Reinforcement Learning

Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Unbiased Methods for Multi-Goal Reinforcement Learning

Goal-Conditioned Q-Learning as Knowledge Distillation

Following Instructions by Imagining and Reaching Visual Goals

Self-Adapting Goals Allow Transfer of Predictive Models to New Tasks

Guided Goal Generation for Hindsight Multi-Goal Reinforcement Learning

Efficient Multi-Goal Reinforcement Learning Via Value Consistency Prioritization

Dissecting Deep RL with High Update Ratios: Combatting Value Divergence

Improvements on Hindsight Learning

Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning

One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL

Visual Navigation with Multiple Goals Based on Deep Reinforcement Learning

Combining Hindsight with Goal-enhanced Prediction for Multi-goal Reinforcement Learning

An Invitation to Deep Reinforcement Learning

Goal exploration augmentation via pre-trained skills for sparse-reward long-horizon goal-conditioned reinforcement learning

Learning with AMIGo: Adversarially Motivated Intrinsic Goals

Hierarchical reinforcement learning with natural language subgoals

Reinforcement Learning with Unsupervised Auxiliary Tasks