Abstract:Artificial intelligence and neuroscience have a long and intertwined history. Advancements in neuroscience research have significantly influenced the development of artificial intelligence systems that have the potential to retain knowledge akin to humans. Building upon foundational insights from neuroscience and existing research in adversarial and continual learning fields, we introduce a novel framework that comprises two key concepts: feature distillation and re-consolidation. The framework distills continual learning (CL) robust features and rehearses them while learning the next task, aiming to replicate the mammalian brain's process of consolidating memories through rehearsing the distilled version of the waking experiences. Furthermore, the proposed framework emulates the mammalian brain's mechanism of memory re-consolidation, where novel experiences influence the assimilation of previous experiences via feature re-consolidation. This process incorporates the new understanding of the CL model after learning the current task into the CL-robust samples of the previous task(s) to mitigate catastrophic forgetting. The proposed framework, called Robust Rehearsal, circumvents the limitations of existing CL frameworks that rely on the availability of pre-trained Oracle CL models to pre-distill CL-robustified datasets for training subsequent CL models. We conducted extensive experiments on three datasets, CIFAR10, CIFAR100, and real-world helicopter attitude datasets, demonstrating that CL models trained using Robust Rehearsal outperform their counterparts' baseline methods. In addition, we conducted a series of experiments to assess the impact of changing memory sizes and the number of tasks, demonstrating that the baseline methods employing robust rehearsal outperform other methods trained without robust rehearsal. Lastly, to shed light on the existence of diverse features, we explore the effects of various optimization training objectives within the realms of joint, continual, and adversarial learning on feature learning in deep neural networks. Our findings indicate that the optimization objective dictates feature learning, which plays a vital role in model performance. Such observation further emphasizes the importance of rehearsing the CL-robust samples in alleviating catastrophic forgetting. In light of our experiments, closely following neuroscience insights can contribute to developing CL approaches to mitigate the long-standing challenge of catastrophic forgetting.

Experience Consistency Distillation Continual Reinforcement Learning for Robotic Manipulation Tasks

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Continual Policy Distillation of Reinforcement Learning-based Controllers for Soft Robotic In-Hand Manipulation

Replay-enhanced Continual Reinforcement Learning

Deep Reinforcement Learning Based Robot Arm Manipulation with Efficient Training Data through Simulation

SLER: Self-generated long-term experience replay for continual reinforcement learning

Reinforcement Learning via Auxiliary Task Distillation

Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal

Stable Continual Reinforcement Learning via Diffusion-based Trajectory Replay

Relay Hindsight Experience Replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewards

Efficient Diversity-based Experience Replay for Deep Reinforcement Learning

Consistent Experience Replay in High-Dimensional Continuous Control with Decayed Hindsights

Soft Hindsight Experience Replay

Relational Experience Replay: Continual Learning by Adaptively Tuning Task-wise Relationship

FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning

ACDER: Augmented Curiosity-Driven Experience Replay

Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control

Brain-Inspired Continual Learning: Robust Feature Distillation and Re-Consolidation for Class Incremental Learning

Effective Interpretable Policy Distillation via Critical Experience Point Identification

Real-time Policy Distillation in Deep Reinforcement Learning

The Ingredients of Real-World Robotic Reinforcement Learning