Abstract:Safety is a critical concern for the next generation of autonomy that is likely to rely heavily on deep neural networks for perception and control. Formally verifying the safety and robustness of well-trained DNNs and learning-enabled systems under attacks, model uncertainties, and sensing errors is essential for safe autonomy. This research proposes a framework to repair unsafe DNNs in safety-critical systems with reachability analysis. The repair process is inspired by adversarial training which has demonstrated high effectiveness in improving the safety and robustness of DNNs. Different from traditional adversarial training approaches where adversarial examples are utilized from random attacks and may not be representative of all unsafe behaviors, our repair process uses reachability analysis to compute the exact unsafe regions and identify sufficiently representative examples to enhance the efficacy and efficiency of the adversarial training. The performance of our framework is evaluated on two types of benchmarks without safe models as references. One is a DNN controller for aircraft collision avoidance with access to training data. The other is a rocket lander where our framework can be seamlessly integrated with the well-known deep deterministic policy gradient (DDPG) reinforcement learning algorithm. The experimental results show that our framework can successfully repair all instances on multiple safety specifications with negligible performance degradation. In addition, to increase the computational and memory efficiency of the reachability analysis algorithm, we propose a depth-first-search algorithm that combines an existing exact analysis method with an over-approximation approach based on a new set representation. Experimental results show that our method achieves a five-fold improvement in runtime and a two-fold improvement in memory usage compared to exact analysis.

ReachNN*: A Tool for Reachability Analysis of Neural-Network Controlled Systems

ReachNN: Reachability Analysis of Neural-Network Controlled Systems.

Reachability Analysis and Safety Verification for Neural Network Control Systems

POLAR-Express: Efficient and Precise Formal Reachability Analysis of Neural-Network Controlled Systems

Scalable Forward Reachability Analysis of Multi-Agent Systems with Neural Network Controllers

NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems

Taming Reachability Analysis of DNN-Controlled Systems via Abstraction-Based Training

Backward Reachability Analysis for Neural Feedback Loops

Exact Imposition of Safety Boundary Conditions in Neural Reachable Tubes

Reachability Analysis for Neural Feedback Systems Using Regressive Polynomial Rule Inference

Neural Network Repair with Reachability Analysis

Reachability analysis for piecewise affine systems with neural network-based controllers

Online Reinforcement Learning Neural Network Controller Design for Nanomanipulation

Contraction-Guided Adaptive Partitioning for Reachability Analysis of Neural Network Controlled Systems

Reachable Set Estimation and Safety Verification for Piecewise Linear Systems with Neural Network Controllers

Approximate Conformance Checking for Closed-Loop Systems With Neural Network Controllers

Hybrid Zonotope-Based Backward Reachability Analysis for Neural Feedback Systems With Nonlinear Plant Models

Efficient Verification of Control Systems with Neural Network Controllers.

Provable Bounds on the Hessian of Neural Networks: Derivative-Preserving Reachability Analysis

Reachability Analysis of a General Class of Neural Ordinary Differential Equations

UR4NNV: Neural Network Verification, Under-approximation Reachability Works!