Abstract:The loss of plasticity in learning agents, analogous to the solidification of neural pathways in biological brains, significantly impedes learning and adaptation in reinforcement learning due to its non-stationary nature. To address this fundamental challenge, we propose a novel approach, Neuroplastic Expansion (NE), inspired by cortical expansion in cognitive science. NE maintains learnability and adaptability throughout the entire training process by dynamically growing the network from a smaller initial size to its full dimension. Our method is designed with three key components: (1) elastic neuron generation based on potential gradients, (2) dormant neuron pruning to optimize network expressivity, and (3) neuron consolidation via experience review to strike a balance in the plasticity-stability dilemma. Extensive experiments demonstrate that NE effectively mitigates plasticity loss and outperforms state-of-the-art methods across various tasks in MuJoCo and DeepMind Control Suite environments. NE enables more adaptive learning in complex, dynamic environments, which represents a crucial step towards transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continually adapting models.

What problem does this paper attempt to address?

The paper attempts to address the issue of agents in Deep Reinforcement Learning (DRL) gradually losing plasticity. This phenomenon is similar to the solidification of neural pathways in the biological brain, severely hindering learning and adaptability, especially in continuous learning tasks in non-stationary environments. Specifically, as the learning process progresses, agents gradually lose the ability to learn from new experiences, which limits their effective learning and response capabilities in complex or non-stationary scenarios. This limitation has become a fundamental obstacle for artificial agents to achieve continuous learning and adaptation. To tackle this challenge, the authors propose a novel approach called Neuroplastic Expansion (NE). Inspired by cortical expansion in cognitive science, NE maintains learning ability and adaptability by dynamically expanding from a small-scale network to a full-size network. The design of NE includes three key components: (1) Elastic neuron generation based on latent gradients; (2) Pruning of dormant neurons to optimize network expressiveness; (3) Neuron consolidation through experience replay to balance the trade-off between plasticity and stability. Through extensive experimental validation, NE effectively mitigates the loss of plasticity and outperforms existing state-of-the-art methods on various tasks in environments such as MuJoCo and DeepMind Control Suite. NE enables agents to learn more flexibly in complex, dynamic environments, which is a crucial step in transitioning deep reinforcement learning from static, one-time training paradigms to more flexible, continuously adaptive models.

Neuroplastic Expansion in Deep Reinforcement Learning

Understanding plasticity in neural networks

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Deep Reinforcement Learning with Plasticity Injection

Plasticity Loss in Deep Reinforcement Learning: A Survey

Differentiable plasticity: training plastic neural networks with backpropagation

Loss of Plasticity in Continual Deep Reinforcement Learning

Disentangling the Causes of Plasticity Loss in Neural Networks

Loss of plasticity in deep continual learning

Learning the Plasticity: Plasticity-Driven Learning Framework in Spiking Neural Networks

Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

Maintaining Plasticity in Deep Continual Learning

Rewiring Neurons in Non-Stationary Environments

Neuron-centric Hebbian Learning

Neural Network Plasticity and Loss Sharpness

A study on the plasticity of neural networks

Balancing complexity, performance and plausibility to meta learn plasticity rules in recurrent spiking networks.

Learning to learn with backpropagation of Hebbian plasticity

Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks

Entropy-based Stability-Plasticity for Lifelong Learning

Evolving interpretable plasticity for spiking networks