Abstract:Underpinning the past decades of work on the design, initialization, and optimization of neural networks is a seemingly innocuous assumption: that the network is trained on a \textit{stationary} data distribution. In settings where this assumption is violated, e.g.\ deep reinforcement learning, learning algorithms become unstable and brittle with respect to hyperparameters and even random seeds. One factor driving this instability is the loss of plasticity, meaning that updating the network's predictions in response to new information becomes more difficult as training progresses. While many recent works provide analyses and partial solutions to this phenomenon, a fundamental question remains unanswered: to what extent do known mechanisms of plasticity loss overlap, and how can mitigation strategies be combined to best maintain the trainability of a network? This paper addresses these questions, showing that loss of plasticity can be decomposed into multiple independent mechanisms and that, while intervening on any single mechanism is insufficient to avoid the loss of plasticity in all cases, intervening on multiple mechanisms in conjunction results in highly robust learning algorithms. We show that a combination of layer normalization and weight decay is highly effective at maintaining plasticity in a variety of synthetic nonstationary learning tasks, and further demonstrate its effectiveness on naturally arising nonstationarities, including reinforcement learning in the Arcade Learning Environment.

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Plasticity Loss in Deep Reinforcement Learning: A Survey

Loss of Plasticity in Continual Deep Reinforcement Learning

Understanding plasticity in neural networks

Deep Reinforcement Learning with Plasticity Injection

Neuroplastic Expansion in Deep Reinforcement Learning

Maintaining Plasticity in Deep Continual Learning

Loss of plasticity in deep continual learning

Neural Network Plasticity and Loss Sharpness

Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

Disentangling the Causes of Plasticity Loss in Neural Networks

PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

A study on the plasticity of neural networks

Self-Normalized Resets for Plasticity in Continual Learning

Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning

Differentiable plasticity: training plastic neural networks with backpropagation

Parseval Regularization for Continual Reinforcement Learning

Maintaining Plasticity in Continual Learning via Regenerative Regularization

Improving Plasticity in Online Continual Learning via Collaborative Learning

Can We Understand Plasticity Through Neural Collapse?