Abstract:Formal verification provides a high degree of confidence in safe system operation, but only if reality matches the verified model. Although a good model will be accurate most of the time, even the best models are incomplete. This is especially true in Cyber-Physical Systems because high-fidelity physical models of systems are expensive to develop and often intractable to verify. Conversely, reinforcement learning-based controllers are lauded for their flexibility in unmodeled environments, but do not provide guarantees of safe operation. This paper presents an approach for provably safe learning that provides the best of both worlds: the exploration and optimization capabilities of learning along with the safety guarantees of formal verification. Our main insight is that formal verification combined with verified runtime monitoring can ensure the safety of a learning agent. Verification results are preserved whenever learning agents limit exploration within the confounds of verified control choices as long as observed reality comports with the model used for off-line verification. When a model violation is detected, the agent abandons efficiency and instead attempts to learn a control strategy that guides the agent to a modeled portion of the state space. We prove that our approach toward incorporating knowledge about safe control into learning systems preserves safety guarantees, and demonstrate that we retain the empirical performance benefits provided by reinforcement learning. We also explore various points in the design space for these justified speculative controllers in a simple model of adaptive cruise control model for autonomous cars.

Formal Control Synthesis Via Safe Reinforcement Learning under Real-Time Specifications

Synthesis of Controllers for Co-Safe Linear Temporal Logic Specifications Using Reinforcement Learning

Safe Reinforcement Learning for Signal Temporal Logic Tasks Using Robust Control Barrier Functions

Safe DNN-type Controller Synthesis for Nonlinear Systems via Meta Reinforcement Learning.

Joint Synthesis of Safety Certificate and Safe Control Policy Using Constrained Reinforcement Learning

Formal synthesis of controllers for safety-critical autonomous systems: Developments and challenges

Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning

Safeguarding Learning-based Control for Smart Energy Systems with Sampling Specifications

Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning

Learning-Based Synthesis of Safety Controllers

Safe Barrier-Constrained Control of Uncertain Systems via Event-triggered Learning

Safe Reinforcement Learning for CPSs via Formal Modeling and Verification

Formal Controller Synthesis for Continuous-Space MDPs via Model-Free Reinforcement Learning

An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems Via Barrier Certificate Generation

Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning

End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Shielded Reinforcement Learning for Hybrid Systems

Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functions

Tunable Reactive Synthesis for Lipschitz-Bounded Systems with Temporal Logic Specifications

Synthesizing Controller for Unsynthesizable Specification Based on Criticality Levels