Abstract:Deep reinforcement learning (RL) has shown promising results in robot motion planning with first attempts in human-robot collaboration (HRC). However, a fair comparison of RL approaches in HRC under the constraint of guaranteed safety is yet to be made. We, therefore, present human-robot gym, a benchmark suite for safe RL in HRC. Our benchmark suite provides eight challenging, realistic HRC tasks in a modular simulation framework. Most importantly, human-robot gym includes a safety shield that provably guarantees human safety. We are, thereby, the first to provide a benchmark suite to train RL agents that adhere to the safety specifications of real-world HRC. This bridges a critical gap between theoretic RL research and its real-world deployment. Our evaluation of six tasks led to three key results: (a) the diverse nature of the tasks offered by human-robot gym creates a challenging benchmark for state-of-the-art RL methods, (b) incorporating expert knowledge in RL training in the form of an action-based reward can outperform the expert, and (c) our agents negligibly overfit to training data.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the issues of safety and task diversity in Human-Robot Collaboration (HRC) using Reinforcement Learning (RL). Specifically: 1. **Safety Issues**: Existing RL applications in HRC are usually limited to narrow task domains and lack effective guarantees for human safety. The paper proposes a benchmark suite called "Human-Robot Gym," which integrates a safety shield (SaRA shield) to ensure the safety of robot behavior during human interaction. 2. **Task Diversity Issues**: Current HRC benchmarks mainly focus on interactions with static humans and lack support for complex dynamic tasks. Human-Robot Gym provides a variety of complex HRC tasks, including object inspection, handover, and collaborative operations, covering different difficulty levels and human dynamics. ### Main Contributions 1. **Safety Shield**: Human-Robot Gym is the first benchmark suite to include a safety shield, which provably ensures human safety. 2. **Diverse Tasks**: It offers a variety of realistic HRC tasks, ranging from simple co-existence tasks to complex collaborative operations. 3. **Modular Simulation Framework**: It provides an extensible simulation framework that supports various robot models and task definitions. 4. **Performance Evaluation**: Through experimental evaluation on 6 tasks, the effectiveness and limitations of RL methods in HRC are validated. ### Experimental Results 1. **Task Diversity**: Experimental results show that some tasks can already be solved by RL (e.g., Reach and Pick and Place), while others still require further improvement (e.g., Collaborative Lifting and Robot-Human Handover). 2. **Importance of Expert Knowledge**: In sparse reward tasks, utilizing expert knowledge (e.g., action-based imitation rewards) can significantly improve the performance of RL agents. For example, in the Pick and Place task, the action-based imitation reward method surpassed the expert policy, achieving nearly 100% success rate. 3. **Overfitting Issue**: Ablation studies reveal that RL agents do not overfit to the limited human motion data during training, indicating that the benchmark suite has good generalization capabilities. ### Conclusion Human-Robot Gym provides a realistic benchmark suite for comparing and evaluating the performance of RL in HRC. By integrating a safety shield and diverse tasks, this suite offers an important tool bridging theoretical research and practical applications.

Human-Robot Gym: Benchmarking Reinforcement Learning in Human-Robot Collaboration

Benchmarking Safe Exploration in Deep Reinforcement Learning

Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark

safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics

Safe Reinforcement Learning in a Simulated Robotic Arm

Provably Safe Deep Reinforcement Learning for Robotic Manipulation in Human Environments

Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention

Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach

Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learning

Learning to be Safe: Deep RL with a Safety Critic

GUARD: A Safe Reinforcement Learning Benchmark

OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research

RobocupGym: A challenging continuous control benchmark in Robocup

A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors

Demonstrating HumanTHOR: A Simulation Platform and Benchmark for Human-Robot Collaboration in a Shared Workspace

Safe Reinforcement Learning Using Black-Box Reachability Analysis

Intrinsically Motivated Multi-Goal Reinforcement Learning Using Robotics Environment Integrated with OpenAI Gym

Benchmarking Reinforcement Learning Methods for Dexterous Robotic Manipulation with a Three-Fingered Gripper

Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning

SOCIALGYM: A Framework for Benchmarking Social Robot Navigation

An Efficient and Responsive Robot Motion Controller for Safe Human-Robot Collaboration