Abstract:One approach to meet the challenges of deep lifelong reinforcement learning (LRL) is careful management of the agent's learning experiences, to learn (without forgetting) and build internal meta-models (of the tasks, environments, agents, and world). Generative replay (GR) is a biologically inspired replay mechanism that augments learning experiences with self-labelled examples drawn from an internal generative model that is updated over time. We present a version of GR for LRL that satisfies two desiderata: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning. In this paper, we study three deep learning architectures for model-free GR, starting from a na\"ive GR and adding ingredients to achieve (a) and (b). We evaluate our proposed algorithms on three different scenarios comprising tasks from the Starcraft-2 and Minigrid domains. We report several key findings showing the impact of the design choices on quantitative metrics that include transfer learning, generalization to unseen tasks, fast adaptation after task change, performance wrt task expert, and catastrophic forgetting. We observe that our GR prevents drift in the features-to-action mapping from the latent vector space of a deep RL agent. We also show improvements in established lifelong learning metrics. We find that a small random replay buffer significantly increases the stability of training. Overall, we find that"hidden replay"(a well-known architecture for class-incremental classification) is the most promising approach that pushes the state-of-the-art in GR for LRL and observe that the architecture of the sleep model might be more important for improving performance than the types of replay used. Our experiments required only 6% of training samples to achieve 80-90% of expert performance in most Starcraft-2 scenarios.

Lifelong Learning using Eigentasks: Task Separation, Skill Acquisition, and Selective Transfer

Active Task Selection for Lifelong Machine Learning

System Design for an Integrated Lifelong Reinforcement Learning Agent for Real-Time Strategy Games

Task-unaware Lifelong Robot Learning with Retrieval-based Weighted Local Adaptation

Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation

Multitask Neuroevolution for Reinforcement Learning with Long and Short Episodes

Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models

Task-Aware Information Routing from Common Representation Space in Lifelong Learning

Statistical Guarantees for Lifelong Reinforcement Learning using PAC-Bayesian Theory

Latent Plans for Task-Agnostic Offline Reinforcement Learning

Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2

Lifelong Dual Generative Adversarial Nets Learning in Tandem

Generative Memory for Lifelong Reinforcement Learning

Lifelong Robot Learning with Human Assisted Language Planners

Lifelong Teacher-Student Network Learning

SECURE: Semantics-aware Embodied Conversation under Unawareness for Lifelong Robot Learning

LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation

Evaluations of the Gap Between Supervised and Reinforcement Lifelong Learning on Robotic Manipulation Tasks

Lifelong Reinforcement Learning via Neuromodulation

Continuous Coordination As a Realistic Scenario for Lifelong Learning

Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning