Abstract:Ideally, we would place a robot in a real-world environment and leave it there improving on its own by gathering more experience autonomously. However, algorithms for autonomous robotic learning have been challenging to realize in the real world. While this has often been attributed to the challenge of sample complexity, even sample-efficient techniques are hampered by two major challenges - the difficulty of providing well "shaped" rewards, and the difficulty of continual reset-free training. In this work, we describe a system for real-world reinforcement learning that enables agents to show continual improvement by training directly in the real world without requiring painstaking effort to hand-design reward functions or reset mechanisms. Our system leverages occasional non-expert human-in-the-loop feedback from remote users to learn informative distance functions to guide exploration while leveraging a simple self-supervised learning algorithm for goal-directed policy learning. We show that in the absence of resets, it is particularly important to account for the current "reachability" of the exploration policy when deciding which regions of the space to explore. Based on this insight, we instantiate a practical learning system - GEAR, which enables robots to simply be placed in real-world environments and left to train autonomously without interruption. The system streams robot experience to a web interface only requiring occasional asynchronous feedback from remote, crowdsourced, non-expert humans in the form of binary comparative feedback. We evaluate this system on a suite of robotic tasks in simulation and demonstrate its effectiveness at learning behaviors both in simulation and the real world. Project website <a class="link-external link-https" href="https://guided-exploration-autonomous-rl.github.io/GEAR/" rel="external noopener nofollow">this https URL</a>.

OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research

Benchmarking Reinforcement Learning Algorithms on Real-World Robots

Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

SOCIALGYM: A Framework for Benchmarking Social Robot Navigation

Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration

RobocupGym: A challenging continuous control benchmark in Robocup

gym-gazebo2, a toolkit for reinforcement learning using ROS 2 and Gazebo

OPEn: An Open-ended Physics Environment for Learning Without a Task

Real World Offline Reinforcement Learning with Realistic Data Source

SocNavGym: A Reinforcement Learning Gym for Social Navigation

Gymnasium: A Standard Interface for Reinforcement Learning Environments

Intrinsically Motivated Multi-Goal Reinforcement Learning Using Robotics Environment Integrated with OpenAI Gym

Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms

SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation

SOCIALGYM 2.0: Simulator for Multi-Agent Social Robot Navigation in Shared Human Spaces

Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo

The Ingredients of Real-World Robotic Reinforcement Learning