Abstract:In Reinforcement Learning, agents learn policies by exploring and interacting with the environment. Due to the curse of dimensionality, learning policies that map high-dimensional sensory input to motor output is particularly challenging. During training, state of the art methods (SAC, PPO, etc.) explore the environment by perturbing the actuation with independent Gaussian noise. While this unstructured exploration has proven successful in numerous tasks, it can be suboptimal for overactuated systems. When multiple actuators, such as motors or muscles, drive behavior, uncorrelated perturbations risk diminishing each other's effect, or modifying the behavior in a task-irrelevant way. While solutions to introduce time correlation across action perturbations exist, introducing correlation across actuators has been largely ignored. Here, we propose LATent TIme-Correlated Exploration (Lattice), a method to inject temporally-correlated noise into the latent state of the policy network, which can be seamlessly integrated with on- and off-policy algorithms. We demonstrate that the noisy actions generated by perturbing the network's activations can be modeled as a multivariate Gaussian distribution with a full covariance matrix. In the PyBullet locomotion tasks, Lattice-SAC achieves state of the art results, and reaches 18% higher reward than unstructured exploration in the Humanoid environment. In the musculoskeletal control environments of MyoSuite, Lattice-PPO achieves higher reward in most reaching and object manipulation tasks, while also finding more energy-efficient policies with reductions of 20-60%. Overall, we demonstrate the effectiveness of structured action noise in time and actuator space for complex motor control tasks. The code is available at: <a class="link-external link-https" href="https://github.com/amathislab/lattice" rel="external noopener nofollow">this https URL</a>.

Structural Parameter Space Exploration for Reinforcement Learning Via a Matrix Variate Distribution

Exploring Parameter Space with Structured Noise for Meta-Reinforcement Learning

Reward Space Noise for Exploration in Deep Reinforcement Learning

Efficient and Scalable Exploration Via Estimation-Error

A Learnable Noise Exploration Method for Multi-Agent Reinforcement Learning

Variational Dynamic for Self-Supervised Exploration in Deep Reinforcement Learning

Random curiosity-driven exploration in deep reinforcement learning

Noise-Regularized Advantage Value for Multi-Agent Reinforcement Learning

Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance

Never Explore Repeatedly in Multi-Agent Reinforcement Learning

Subspace-Aware Exploration for Sparse-Reward Multi-Agent Tasks.

Adaptive Exploration Network Policy for Effective Exploration in Reinforcement Learning

Exploration in Feature Space for Reinforcement Learning

Active exploration in parameterized reinforcement learning

Latent Exploration for Reinforcement Learning

Multimodal Reward Shaping for Efficient Exploration in Reinforcement Learning

Deterministic Exploration via Stationary Bellman Error Maximization

Noisy Spiking Actor Network for Exploration

Dynamic Bottleneck for Robust Self-Supervised Exploration

PP-PG: Combining Parameter Perturbation with Policy Gradient Methods for Effective and Efficient Explorations in Deep Reinforcement Learning

Effective Exploration Based on the Structural Information Principles