Abstract:Safety is a critical concern for the next generation of autonomy that is likely to rely heavily on deep neural networks for perception and control. Formally verifying the safety and robustness of well-trained DNNs and learning-enabled systems under attacks, model uncertainties, and sensing errors is essential for safe autonomy. This research proposes a framework to repair unsafe DNNs in safety-critical systems with reachability analysis. The repair process is inspired by adversarial training which has demonstrated high effectiveness in improving the safety and robustness of DNNs. Different from traditional adversarial training approaches where adversarial examples are utilized from random attacks and may not be representative of all unsafe behaviors, our repair process uses reachability analysis to compute the exact unsafe regions and identify sufficiently representative examples to enhance the efficacy and efficiency of the adversarial training. The performance of our framework is evaluated on two types of benchmarks without safe models as references. One is a DNN controller for aircraft collision avoidance with access to training data. The other is a rocket lander where our framework can be seamlessly integrated with the well-known deep deterministic policy gradient (DDPG) reinforcement learning algorithm. The experimental results show that our framework can successfully repair all instances on multiple safety specifications with negligible performance degradation. In addition, to increase the computational and memory efficiency of the reachability analysis algorithm, we propose a depth-first-search algorithm that combines an existing exact analysis method with an over-approximation approach based on a new set representation. Experimental results show that our method achieves a five-fold improvement in runtime and a two-fold improvement in memory usage compared to exact analysis.

nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems

Application of the Neural Network Dependability Kit in Real-World Environments

Dependable Neural Networks for Safety Critical Tasks

There is Limited Correlation Between Coverage and Robustness for Deep Neural Networks

Analyzing neural network behavior through deep statistical model checking

Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control

PNNUAD: Perception Neural Networks Uncertainty Aware Decision-Making for Autonomous Vehicle

DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars

Evaluating Deep Neural Networks in Deployment (A Comparative and Replicability Study)

NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems

Artificial Neural Networks for Space and Safety-Critical Applications: Reliability Issues and Potential Solutions

Engineering Reliable Deep Learning Systems

Deep Learning Safety Concerns in Automated Driving Perception

Inherent Diverse Redundant Safety Mechanisms for AI-based Software Elements in Automotive Applications

Deep Statistical Model Checking

Reliability Assurance for Deep Neural Network Architectures Against Numerical Defects

Verification and Design of Robust and Safe Neural Network-enabled Autonomous Systems

Automatic AI controller that can drive with confidence: steering vehicle with uncertainty knowledge

Neural Network Repair with Reachability Analysis

SoK: Certified Robustness for Deep Neural Networks

Reliability of Deep Neural Networks for an End-to-End Imitation Learning-Based Lane Keeping