Abstract:Safety is a critical concern when applying reinforcement learning (RL) to real-world control tasks. However, existing safe RL works either only consider expected safety constraint violations and fail to maintain safety guarantees, or use overly conservative safety certificate tools borrowed from safe control theory, which sacrifices reward optimization and relies on analytic system models. This letter proposes a model-free safe RL algorithm that achieves near-zero constraint violations with high rewards. Our key idea is to jointly learn a policy and a neural barrier certificate under stepwise state constraint setting. The barrier certificate is learned in a model-free manner by minimizing the violations of appropriate barrier properties on transition data collected by the policy. We extend the single-step invariant property of the barrier certificate to a multi-step version and construct the corresponding multi-step invariant loss. This loss balances the bias and variance of the barrier certificate and enhances both the safety and performance of the policy. The policy is optimized under the constraint of the multi-step invariant property using the Lagrangian method. We optimize the policy in a model-free manner by introducing an importance sampling weight in the constraint. We test our algorithm on multiple problems, including classic control tasks, robot collision avoidance, and autonomous driving. Results show that our algorithm achieves near-zero constraint violations and high performance compared to the baselines. Moreover, the learned barrier certificates successfully identify the feasible regions on multiple tasks.

Safe Control With Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction Methods for Robotics and Control

Learning Observation-Based Certifiable Safe Policy for Decentralized Multi-Robot Navigation

Learning Safe, Generalizable Perception-Based Hybrid Control With Certificates

Learning Safe Control for Multi-Robot Systems: Methods, Verification, and Open Challenges

SABLAS: Learning Safe Control for Black-box Dynamical Systems

Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions

Learning Control Barrier Functions and their application in Reinforcement Learning: A Survey

Learning safe neural network controllers with barrier certificates

Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning

Learning-Based Safe Control for Robot and Autonomous Vehicle Using Efficient Safety Certificate

Learning for Safety-Critical Control with Control Barrier Functions

Myopically Verifiable Probabilistic Certificates for Safe Control and Learning

Safe Online Dynamics Learning with Initially Unknown Models and Infeasible Safety Certificates

Learning Certified Control using Contraction Metric

Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions

Formally Verifying Deep Reinforcement Learning Controllers with Lyapunov Barrier Certificates

Transfer of Safety Controllers Through Learning Deep Inverse Dynamics Model

Model-Free Safe Reinforcement Learning Through Neural Barrier Certificate

Safety-Aware Learning-Based Control of Systems with Uncertainty Dependent Constraints (extended version)

Learning-based Model Predictive Control for Safe Exploration and Reinforcement Learning