PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

Hojoon Lee,Hanseul Cho,Hyunseung Kim,Daehoon Gwak,Joonkee Kim,Jaegul Choo,Se-Young Yun,Chulhee Yun

2023-12-09

Abstract:In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms can improve sample efficiency by allowing multiple updates per environment interaction. However, these multiple updates often lead the model to overfit to earlier interactions, which is referred to as the loss of plasticity. Our study investigates the underlying causes of this phenomenon by dividing plasticity into two aspects. Input plasticity, which denotes the model's adaptability to changing input data, and label plasticity, which denotes the model's adaptability to evolving input-output relationships. Synthetic experiments on the CIFAR-10 dataset reveal that finding smoother minima of loss landscape enhances input plasticity, whereas refined gradient propagation improves label plasticity. Leveraging these findings, we introduce the PLASTIC algorithm, which harmoniously combines techniques to address both concerns. With minimal architectural modifications, PLASTIC achieves competitive performance on benchmarks including Atari-100k and Deepmind Control Suite. This result emphasizes the importance of preserving the model's plasticity to elevate the sample efficiency in RL. The code is available at <a class="link-external link-https" href="https://github.com/dojeon-ai/plastic" rel="external noopener nofollow">this https URL</a>.

Machine Learning

What problem does this paper attempt to address?

This paper attempts to address the issue of improving sample efficiency in Reinforcement Learning (RL), particularly in situations where data acquisition is costly and risky. Specifically, the paper explores how to enhance the plasticity of the model to improve sample efficiency. The authors divide plasticity into two aspects: input plasticity and label plasticity. Input plasticity refers to the model's ability to adapt to changes in input data, while label plasticity refers to the model's ability to adapt to the evolution of input-output relationships. The paper investigates the impact of loss surface smoothness and gradient propagation on these two types of plasticity through the design of synthetic experiments and proposes a new algorithm named PLASTIC. PLASTIC combines multiple techniques aimed at simultaneously improving input plasticity and label plasticity. Experimental results show that PLASTIC performs excellently in benchmark tests such as Atari-100k and DeepMind Control Suite, demonstrating its effectiveness in improving RL sample efficiency.

PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning

Deep Reinforcement Learning with Plasticity Injection

Plasticity Loss in Deep Reinforcement Learning: A Survey

Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Understanding plasticity in neural networks

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Neuroplastic Expansion in Deep Reinforcement Learning

Slow and Steady Wins the Race: Maintaining Plasticity with Hare and Tortoise Networks

Sample Efficient Deep Reinforcement Learning with Online State Abstraction and Causal Transformer Model Prediction

Disentangling the Causes of Plasticity Loss in Neural Networks

Loss of Plasticity in Continual Deep Reinforcement Learning

Differentiable plasticity: training plastic neural networks with backpropagation

PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics.

Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Neural Network Plasticity and Loss Sharpness

Improving Plasticity in Online Continual Learning via Collaborative Learning

Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs

Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning

PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning

Achieving efficient interpretability of reinforcement learning via policy distillation and selective input gradient regularization