Abstract:A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the reinforcement learning (RL) agent's behavior as the environment changes over its lifetime while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, that is, dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process (CRP) prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of stability, overall performance, and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.

Discovering Denial Constraints Based on Deep Reinforcement Learning

Discovering Editing Rules by Deep Reinforcement Learning.

An Adaptive Framework for Confidence-constraint Rule Set Learning Algorithm in Large Dataset

DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning

Effective Reinforcement Learning Based on Structural Information Principles

A Benchmark Study of Deep-RL Methods for Maximum Coverage Problems over Graphs

Dynamics-Adaptive Continual Reinforcement Learning Via Progressive Contextualization.

Safe Reinforcement Learning via Hierarchical Adaptive Chance-Constraint Safeguards

Constraint-Adaptive Rule Mining in Large Databases

Resilient Constrained Reinforcement Learning

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

DeLF: Designing Learning Environments with Foundation Models

Diversifying repairs of Denial constraint violations

Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training

Learning What to Defer for Maximum Independent Sets

Incremental discovery of denial constraints

CDT: Cascading Decision Trees for Explainable Reinforcement Learning

Bridging the gap between Markowitz planning and deep reinforcement learning