Abstract:Safety is the major consideration in controlling complex dynamical systems using reinforcement learning (RL), where the safety certificate can provide provable safety guarantee. A valid safety certificate is an energy function indicating that safe states are with low energy, and there exists a corresponding safe control policy that allows the energy function to always dissipate. The safety certificate and the safe control policy are closely related to each other and both challenging to synthesize. Therefore, existing learning-based studies treat either of them as prior knowledge to learn the other, which limits their applicability with general unknown dynamics. This paper proposes a novel approach that simultaneously synthesizes the energy-function-based safety certificate and learns the safe control policy with CRL. We do not rely on prior knowledge about either an available model-based controller or a perfect safety certificate. In particular, we formulate a loss function to optimize the safety certificate parameters by minimizing the occurrence of energy increases. By adding this optimization procedure as an outer loop to the Lagrangian-based constrained reinforcement learning (CRL), we jointly update the policy and safety certificate parameters and prove that they will converge to their respective local optima, the optimal safe policy and a valid safety certificate. We evaluate our algorithms on multiple safety-critical benchmark environments. The results show that the proposed algorithm learns provably safe policies with no constraint violation. The validity or feasibility of synthesized safety certificate is also verified numerically.

Hybrid Controller Synthesis for Nonlinear Systems Subject to Reach-Avoid Constraints.

Safe DNN-type Controller Synthesis for Nonlinear Systems via Meta Reinforcement Learning.

Optimal Control for Constrained Discrete-Time Nonlinear Systems Based on Safe Reinforcement Learning.

An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems Via Barrier Certificate Generation

Formal controller synthesis for hybrid systems using genetic programming

Scalable Synthesis of Verified Controllers in Deep Reinforcement Learning

Provable Reach-avoid Controllers Synthesis Based on Inner-approximating Controlled Reach-avoid Sets

Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions

Correct-by-Construction for Hybrid Systems by Synthesizing Reset Controller

Systematic, Lyapunov-Based, Safe and Stabilizing Controller Synthesis for Constrained Nonlinear Systems

Learning safe neural network controllers with barrier certificates

Reset Controller Synthesis by Reach-avoid Analysis for Delay Hybrid Systems

Synthesis and verification of robust-adaptive safe controllers

Shielded Reinforcement Learning for Hybrid Systems

Joint Synthesis of Safety Certificate and Safe Control Policy Using Constrained Reinforcement Learning

Lyapunov-stable neural-network control

Controller Synthesis for Linear Time-varying Systems with Adversaries

Designing System Level Synthesis Controllers for Nonlinear Systems with Stability Guarantees

Online Learning and Control Synthesis for Reachable Paths of Unknown Nonlinear Systems

A framework of learning controller with Lyapunov-based constraint and application

Funnel-based Control for Reach-Avoid-Stay Specifications