Abstract:Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. Various techniques have been proposed to improve performance in goal-conditioned environments, such as automatic curriculum generation and goal relabeling. In this work, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. In particular: the current Q-value function and the target Q-value estimate are both functions of the goal, and we would like to train the Q-value function to match its target for all goals. We therefore apply Gradient-Based Attention Transfer (Zagoruyko and Komodakis 2017), a knowledge distillation technique, to the Q-function update. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals, where the agent can attain a reward by achieving any one of a large set of objectives, all specified at test time. Finally, to provide theoretical support, we give examples of classes of environments where (under some assumptions) standard off-policy algorithms such as DDPG require at least O(d^2) replay buffer transitions to learn an optimal policy, while our proposed technique requires only O(d) transitions, where d is the dimensionality of the goal and state space. Code is available at <a class="link-external link-https" href="https://github.com/alevine0/ReenGAGE" rel="external noopener nofollow">this https URL</a>.

Goal-oriented Knowledge Reuse Via Curriculum Evolution for Reinforcement Learning-based Adaptation

Failure-aware Policy Learning for Self-assessable Robotics Tasks

Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution

Evolving Curricula with Regret-Based Environment Design

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

Self-Adapting Goals Allow Transfer of Predictive Models to New Tasks

KnowRU: Knowledge Reuse Via Knowledge Distillation in Multi-Agent Reinforcement Learning

Learning by Reusing Previous Advice in Teacher-Student Paradigm.

Teacher-student curriculum learning for reinforcement learning

Heuristically Adaptive Policy Reuse in Reinforcement Learning

Learning Curriculum Policies for Reinforcement Learning

Knowledge Guided Fuzzy Deep Reinforcement Learning

KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning

Knowledge Reuse of Multi-Agent Reinforcement Learning in Cooperative Tasks

Goal-Conditioned Q-Learning as Knowledge Distillation

Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards

Combining Model-Based $Q$ -Learning with Structural Knowledge Transfer for Robot Skill Learning

SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration

Generating Automatic Curricula via Self-Supervised Active Domain Randomization

TendencyRL: Multi-stage Discriminative Hints for Efficient Goal-Oriented Reverse Curriculum Learning.

Sample-Efficient Curriculum Reinforcement Learning for Complex Reward Functions