Abstract:Ensuring safety of nonlinear systems under model uncertainty and external disturbances is crucial, especially for real-world control tasks. Predictive methods such as robust model predictive control (RMPC) require solving nonconvex optimization problems online, which leads to high computational burden and poor scalability. Reinforcement learning (RL) works well with complex systems, but pays the price of losing rigorous safety guarantee. This paper presents a theoretical framework that bridges the advantages of both RMPC and RL to synthesize safety filters for nonlinear systems with state- and action-dependent uncertainty. We decompose the robust invariant set (RIS) into two parts: a target set that aligns with terminal region design of RMPC, and a reach-avoid set that accounts for the rest of RIS. We propose a policy iteration approach for robust reach-avoid problems and establish its monotone convergence. This method sets the stage for an adversarial actor-critic deep RL algorithm, which simultaneously synthesizes a reach-avoid policy network, a disturbance policy network, and a reach-avoid value network. The learned reach-avoid policy network is utilized to generate nominal trajectories for online verification, which filters potentially unsafe actions that may drive the system into unsafe regions when worst-case disturbances are applied. We formulate a second-order cone programming (SOCP) approach for online verification using system level synthesis, which optimizes for the worst-case reach-avoid value of any possible trajectories. The proposed safety filter requires much lower computational complexity than RMPC and still enjoys persistent robust safety guarantee. The effectiveness of our method is illustrated through a numerical example.

Safe Reinforcement Learning Filter for Multicopter Collision-Free Tracking under disturbances

Safe Tracking Control for Free-Flying Space Robots Via Control Barrier Functions

Safe and Efficient Reinforcement Learning Using Disturbance-Observer-Based Control Barrier Functions

Safe Reinforcement Learning for Model-Reference Trajectory Tracking of Uncertain Autonomous Vehicles with Model-Based Acceleration

Learning Predictive Safety Filter via Decomposition of Robust Invariant Set

Ensuring Safety in Target Pursuit Control: A CBF-Safe Reinforcement Learning Approach

Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning

Safety Filtering While Training: Improving the Performance and Sample Efficiency of Reinforcement Learning Agents

Reinforcement Learning‐based Tracking Control for a Quadrotor Unmanned Aerial Vehicle under External Disturbances

Conformal Predictive Safety Filter for RL Controllers in Dynamic Environments

Robust Reinforcement Learning with UUB Guarantee for Safe Motion Control of Autonomous Robots

Lyapunov Constrained Safe Reinforcement Learning for Multicopter Visual Servoing

Q-learning-based Model-free Safety Filter

Reachability-Based Trajectory Safeguard (RTS): A Safe and Fast Reinforcement Learning Safety Layer for Continuous Control

SSRL: A Safe and Smooth Reinforcement Learning Approach for Collision Avoidance in Navigation

Safe Returning FaSTrack with Robust Control Lyapunov-Value Functions

Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

SRL-TR2: A Safe Reinforcement Learning Based TRajectory TRacker Framework

Reinforcement Learning Control Strategy of Quadrotor Unmanned Aerial Vehicles Based on Linear Filter

Reinforcement learning-based finite-time tracking control of an unknown unmanned surface vehicle with input constraints