Abstract:A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the reinforcement learning (RL) agent's behavior as the environment changes over its lifetime while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, that is, dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process (CRP) prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of stability, overall performance, and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.

Deep Reinforcement Learning with Explicit Context Representation

IReCa: Intrinsic Reward-enhanced Context-aware Reinforcement Learning for Human-AI Coordination

Dynamics-Adaptive Continual Reinforcement Learning Via Progressive Contextualization.

Retrieval-Augmented Decision Transformer: External Memory for In-context RL

Reinforcement Learning Approach for Integrating Compressed Contexts into Knowledge Graphs

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Episodic Reinforcement Learning with Expanded State-reward Space

Meta-Inverse Reinforcement Learning with Probabilistic Context Variables

Dynamics-Aware Context Representation for Domain Adaptation in Reinforcement Learning

In-context Exploration-Exploitation for Reinforcement Learning

Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation

Reinforcement Learning with History-Dependent Dynamic Contexts

Deep Reinforcement Learning with Causality-based Intrinsic Reward

Deep Reinforcement Learning with Decorrelation

From Bandits Model to Deep Deterministic Policy Gradient, Reinforcement Learning with Contextual Information

Achieving Safe Deep Reinforcement Learning Via Environment Comprehension Mechanism

DeRL: Coupling Decomposition in Action Space for Reinforcement Learning Task

A probabilistic successor representation for context-dependent learning.

Contextualize Me -- The Case for Context in Reinforcement Learning

Latent Context Based Soft Actor-Critic

DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstruction