Abstract:A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the reinforcement learning (RL) agent's behavior as the environment changes over its lifetime while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, that is, dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process (CRP) prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of stability, overall performance, and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.

Context-Dependent Structure Control for Adaptive Behavior Selection

Adaptive Locomotion Control of a Hexapod Robot Via Bio-Inspired Learning

Active Structures Integrated with Wireless Sensor and Actuator Networks: a Bio-Inspired Control Framework

Behavior Selection Based on Conditioned Reflex and Emotion

Modeling and Control Architecture for the Competitive Networked Robot System Based on POMDP

Hierarchical learning control for autonomous robots inspired by central nervous system

Hierarchical and State-based Architectures for Robot Behavior Planning and Control

Universal Morphology Control via Contextual Modulation

A Robotic Cognitive Control Framework for Collaborative Task Execution and Learning

Deliberative/Reactive Architecture of a Multi-Robot Patrol System Based on Supervisory Control Theory

Structure and Control of Biology-inspired Networks

A Hybrid Systems-based Hierarchical Control Architecture for Heterogeneous Field Robot Teams

Human-Robot Mutual Adaptation in Shared Autonomy

A Control Scheme for Smooth Transition in Physical Human-Robot-Environment Between Two Modes: Augmentation and Autonomous

Hierarchical Control and Learning of a Foraging CyberOctopus

An Optimization Approach for a Robust and Flexible Control in Collaborative Applications

Dynamics-Adaptive Continual Reinforcement Learning Via Progressive Contextualization.

Distributed Hierarchical Shared Control for Flexible Multirobot Maneuver Through Dense Undetectable Obstacles

C3F: Constant Collaboration and Communication Framework for Graph-Representation Dynamic Multi-Robotic Systems

Self-organized adaptation of a simple neural circuit enables complex robot behaviour