Abstract:Reinforcement Learning (RL) has demonstrated remarkable success across various domains. Nonetheless, a significant challenge in RL is to ensure safety, particularly when deploying it in safety-critical applications such as robotics and autonomous driving. In this work, we develop a robust and safe RL methodology grounded in manifold space. Initially, we construct a constrained manifold space, taking safety constraints into consideration. We then propose a robust safe RL approach, supported by theoretical analysis, based on the value at risk and conditional value at risk, in order to enhance the robustness of safety. Our methodology is designed to ensure safety within stochastic constraint environments. Following the theoretical analysis, we develop a practical, safe algorithm to search for a robust safe policy on stochastic constraint manifolds (ROSCOM). We evaluate the effectiveness of our approach through circular motion and air-hockey tasks. Our experiments demonstrate that ROSCOM outperforms existing baselines in terms of both reward and safety. Note to Practitioners-Real-world applications often involve inherent uncertainties, noise, and high-dimensional spaces. This complexity accentuates the urgency and challenge of ensuring safety in robot learning, especially when implementing RL in practical environments. To address this critical issue, we build a stochastic constraint manifold to delineate the safety space, thus establishing a rigorous framework for robot learning at each iteration. Compared with state-of-the-art baselines, our method can provide remarkable performance regarding safety and reward performance. For example, in an air hockey robot learning task, our method has demonstrated a remarkable $50\%$ enhancement in safety performance compared to the ATACOM framework, while concurrently exhibiting superior reward performance. Moreover, in contrast to traditional algorithms, including CPO, PCPO, our method has achieved a 99% improvement in safety performance, coupled with significantly superior reward performance. These empirical insights render our approach not only theoretically sound but also practically efficacious, indicating its potential as a useful tool in real robot learning and beyond.

Space Processor Computation Time Analysis for Reinforcement Learning and Run Time Assurance Control Policies

Run Time Assured Reinforcement Learning for Six Degree-of-Freedom Spacecraft Inspection

Demonstrating Reinforcement Learning and Run Time Assurance for Spacecraft Inspection Using Unmanned Aerial Vehicles

Safe Spacecraft Inspection via Deep Reinforcement Learning and Discrete Control Barrier Functions

Run Time Assurance for Autonomous Spacecraft Inspection

Systems Theoretic Process Analysis of a Run Time Assured Neural Network Control System

Deep Reinforcement Learning for Scalable Multiagent Spacecraft Inspection

ROSCOM: Robust Safe Reinforcement Learning on Stochastic Constraint Manifolds

Investigating the Impact of Choice on Deep Reinforcement Learning for Space Controls

Comparing Run Time Assurance Approaches for Safe Spacecraft Docking

Run Time Assurance for Simultaneous Constraint Satisfaction During Spacecraft Attitude Maneuvering

Deployable Reinforcement Learning with Variable Control Rate

Runtime Safety Assurance Using Reinforcement Learning

Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach

Reinforcement Learning Based on Real-Time Iteration NMPC

Training in Task Space to Speed Up and Guide Reinforcement Learning

Searching for Optimal Runtime Assurance via Reachability and Reinforcement Learning

Fixed-Time Fault-Tolerant Optimal Attitude Control of Spacecraft With Performance Constraint Via Reinforcement Learning

Control of Small Spacecraft by Optimal Output Regulation: A Reinforcement Learning Approach

Model-Based Safe Reinforcement Learning With Time-Varying Constraints: Applications to Intelligent Vehicles

A Plug-and-Play Fully On-the-Job Real-Time Reinforcement Learning Algorithm for a Direct-Drive Tandem-Wing Experiment Platforms Under Multiple Random Operating Conditions