Abstract:In recent years, significant progress has been made in the field of robotic reinforcement learning (RL), enabling methods that handle complex image observations, train in the real world, and incorporate auxiliary data, such as demonstrations and prior experience. However, despite these advances, robotic RL remains hard to use. It is acknowledged among practitioners that the particular implementation details of these algorithms are often just as important (if not more so) for performance as the choice of algorithm. We posit that a significant challenge to widespread adoption of robotic RL, as well as further development of robotic RL methods, is the comparative inaccessibility of such methods. To address this challenge, we developed a carefully implemented library containing a sample efficient off-policy deep RL method, together with methods for computing rewards and resetting the environment, a high-quality controller for a widely-adopted robot, and a number of challenging example tasks. We provide this library as a resource for the community, describe its design choices, and present experimental results. Perhaps surprisingly, we find that our implementation can achieve very efficient learning, acquiring policies for PCB board assembly, cable routing, and object relocation between 25 to 50 minutes of training per policy on average, improving over state-of-the-art results reported for similar tasks in the literature. These policies achieve perfect or near-perfect success rates, extreme robustness even under perturbations, and exhibit emergent recovery and correction behaviors. We hope that these promising results and our high-quality open-source implementation will provide a tool for the robotics community to facilitate further developments in robotic RL. Our code, documentation, and videos can be found at

Tianshou: A Highly Modularized Deep Reinforcement Learning Library

XuanCe: A Comprehensive and Unified Deep Reinforcement Learning Library

rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch

TorchRL: A data-driven decision-making library for PyTorch

ChainerRL: A Deep Reinforcement Learning Library

RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms

SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning

Open-Source Reinforcement Learning Environments Implemented in MuJoCo with Franka Manipulator

DeepMind Control Suite

RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark

CytonRL: an Efficient Reinforcement Learning Open-source Toolkit Implemented in C++

Optimizing Communication in Deep Reinforcement Learning with XingTian

Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform

Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

OpenSLU: A Unified, Modularized, and Extensible Toolkit for Spoken Language Understanding

Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms

RLtools: A Fast, Portable Deep Reinforcement Learning Library for Continuous Control

Transformer in Transformer As Backbone for Deep Reinforcement Learning

Tactics2D: A Multi-agent Reinforcement Learning Environment for Driving Decision-making

MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library

Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban