Abstract:A major challenge in the Deep RL (DRL) community is to train agents able to generalize over unseen situations, which is often approached by training them on a diversity of tasks (or environments). A powerful method to foster diversity is to procedurally generate tasks by sampling their parameters from a multi-dimensional distribution, enabling in particular to propose a different task for each training episode. In practice, to get the high diversity of training tasks necessary for generalization, one has to use complex procedural generation systems. With such generators, it is hard to get prior knowledge on the subset of tasks that are actually learnable at all (many generated tasks may be unlearnable), what is their relative difficulty and what is the most efficient task distribution ordering for training. A typical solution in such cases is to rely on some form of Automated Curriculum Learning (ACL) to adapt the sampling distribution. One limit of current approaches is their need to explore the task space to detect progress niches over time, which leads to a loss of time. Additionally, we hypothesize that the induced noise in the training data may impair the performances of brittle DRL learners. We address this problem by proposing a two stage ACL approach where 1) a teacher algorithm first learns to train a DRL agent with a high-exploration curriculum, and then 2) distills learned priors from the first run to generate an "expert curriculum" to re-train the same agent from scratch. Besides demonstrating 50% improvements on average over the current state of the art, the objective of this work is to give a first example of a new research direction oriented towards refining ACL techniques over multiple learners, which we call Classroom Teaching.

Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

On Recursive Temporal Difference and Eligibility Traces

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

Expected Eligibility Traces

TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning

Deep Reinforcement Learning in Nonstationary Environments With Unknown Change Points

Parallel reinforcement learning with eligibility traces

Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learning

Real-Time Recurrent Learning using Trace Units in Reinforcement Learning

Data-efficient Deep Reinforcement Learning Method Toward Scaling Continuous Robotic Task with Sparse Rewards.

Efficient Online Reinforcement Learning with Offline Data

Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning

Metatrace: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control

Episodic Reinforcement Learning with Expanded State-reward Space

Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning

Dynamic Sparse Training for Deep Reinforcement Learning

Exploring the Promise and Limits of Real-Time Recurrent Learning