Abstract:Through many recent successes in simulation, model-free reinforcement learning has emerged as a promising approach to solving continuous control robotic tasks. The research community is now able to reproduce, analyze and build quickly on these results due to open source implementations of learning algorithms and simulated benchmark tasks. To carry forward these successes to real-world applications, it is crucial to withhold utilizing the unique advantages of simulations that do not transfer to the real world and experiment directly with physical robots. However, reinforcement learning research with physical robots faces substantial resistance due to the lack of benchmark tasks and supporting source code. In this work, we introduce several reinforcement learning tasks with multiple commercially available robots that present varying levels of learning difficulty, setup, and repeatability. On these tasks, we test the learning performance of off-the-shelf implementations of four reinforcement learning algorithms and analyze sensitivity to their hyper-parameters to determine their readiness for applications in various real-world tasks. Our results show that with a careful setup of the task interface and computations, some of these implementations can be readily applicable to physical robots. We find that state-of-the-art learning algorithms are highly sensitive to their hyper-parameters and their relative ordering does not transfer across tasks, indicating the necessity of re-tuning them for each task for best performance. On the other hand, the best hyper-parameter configuration from one task may often result in effective learning on held-out tasks even with different robots, providing a reasonable default. We make the benchmark tasks publicly available to enhance reproducibility in real-world reinforcement learning.

Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

Benchmarking Deep Reinforcement Learning for Continuous Control

Deep Reinforcement Learning That Matters

Benchmarking Reinforcement Learning Algorithms on Real-World Robots

Investigating Generalisation in Continuous Deep Reinforcement Learning

Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies

Optimal Control-Based Baseline for Guided Exploration in Policy Gradient Methods

The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations

A survey of benchmarking frameworks for reinforcement learning

RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control

Continuous control with deep reinforcement learning

Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control

Position: Benchmarking is Limited in Reinforcement Learning Research

Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers

Policy ensemble gradient for continuous control problems in deep reinforcement learning

Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning

Off-Policy Deep Reinforcement Learning without Exploration

Benchmarking Safe Exploration in Deep Reinforcement Learning

An Optical Control Environment for Benchmarking Reinforcement Learning Algorithms

The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

An efficient reinforcement learning algorithm for learning deterministic policies in continuous domains