Abstract:Despite the tremendous advances that have been made in the last decade on developing useful machine-learning applications, their wider adoption has been hindered by the lack of strong assurance guarantees that can be made about their behavior. In this paper, we consider how formal verification techniques developed for traditional software systems can be repurposed for verification of reinforcement learning-enabled ones, a particularly important class of machine learning systems. Rather than enforcing safety by examining and altering the structure of a complex neural network implementation, our technique uses blackbox methods to synthesizes deterministic programs, simpler, more interpretable, approximations of the network that can nonetheless guarantee desired safety properties are preserved, even when the network is deployed in unanticipated or previously unobserved environments. Our methodology frames the problem of neural network verification in terms of a counterexample and syntax-guided inductive synthesis procedure over these programs. The synthesis procedure searches for both a deterministic program and an inductive invariant over an infinite state transition system that represents a specification of an application's control logic. Additional specifications defining environment-based constraints can also be provided to further refine the search space. Synthesized programs deployed in conjunction with a neural network implementation dynamically enforce safety conditions by monitoring and preventing potentially unsafe actions proposed by neural policies. Experimental results over a wide range of cyber-physical applications demonstrate that software-inspired formal verification techniques can be used to realize trustworthy reinforcement learning systems with low overhead.

Formal Synthesis of Neural Barrier Certificates for Continuous Systems Via Counterexample Guided Learning

Automated and Formal Synthesis of Neural Barrier Certificates for Dynamical Models

Synthesizing Barrier Certificates Using Neural Networks.

Synthesizing Barrier Certificates of Neural Network Controlled Continuous Systems Via Approximations.

Polynomial Neural Barrier Certificate Synthesis of Hybrid Systems Via Counterexample Guidance

Verifying Neural Network Controlled Systems Using Neural Networks

An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems Via Barrier Certificate Generation

Simultaneous Synthesis and Verification of Neural Control Barrier Functions through Branch-and-Bound Verification-in-the-loop Training

Learning safe neural network controllers with barrier certificates

Synthesizing ReLU Neural Networks with Two Hidden Layers As Barrier Certificates for Hybrid Systems

Synthesizing Invariant Barrier Certificates via Difference-of-Convex Programming

Verification-Aided Learning of Neural Network Barrier Functions with Termination Guarantees

A General Framework for Verification and Control of Dynamical Models via Certificate Synthesis

Accelerated Synthesis of Neural Network-based Barrier Certificates Using Collaborative Learning

A General Verification Framework for Dynamical and Control Models via Certificate Synthesis

Safety Certification for Stochastic Systems via Neural Barrier Functions

Exponential-Condition-Based Barrier Certificate Generation for Safety Verification of Hybrid Systems

A New Barrier Certificate for Safety Verification of Hybrid Systems

Learning a Formally Verified Control Barrier Function in Stochastic Environment

An Inductive Synthesis Framework for Verifiable Reinforcement Learning

Barrier certificates revisited