Abstract:This technical report presents research results achieved in the field of verification of trained Convolutional Neural Network (CNN) used for image classification in safety-critical applications. As running example, we use the obstacle detection function needed in future autonomous freight trains with Grade of Automation (GoA) 4. It is shown that systems like GoA 4 freight trains are indeed certifiable today with new standards like ANSI/UL 4600 and ISO 21448 used in addition to the long-existing standards EN 50128 and EN 50129. Moreover, we present a quantitative analysis of the system-level hazard rate to be expected from an obstacle detection function. It is shown that using sensor/perceptor fusion, the fused detection system can meet the tolerable hazard rate deemed to be acceptable for the safety integrity level to be applied (SIL-3). A mathematical analysis of CNN models is performed which results in the identification of classification clusters and equivalence classes partitioning the image input space of the CNN. These clusters and classes are used to introduce a novel statistical testing method for determining the residual error probability of a trained CNN and an associated upper confidence limit. We argue that this greybox approach to CNN verification, taking into account the CNN model's internal structure, is essential for justifying that the statistical tests have covered the trained CNN with its neurons and inter-layer mappings in a comprehensive way.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to verify and confirm the reliability of the image classification system based on convolutional neural network (CNN) in a highly automated train control system, especially in terms of the obstacle detection function. Specifically, the paper focuses on the obstacle detection system in future autonomous freight trains (automation level 4, GoA 4), which operate without drivers and other support personnel. The main objectives of the paper include: 1. **Certification issues**: - The paper explores how to certify AI - based obstacle detection systems through new standards (such as ANSI/UL 4600 and ISO 21448), which supplement the existing EN 50128 and EN 50129 standards. - A new system architecture is proposed, in which most safety - critical control components can still be verified, validated and certified by traditional methods, while the obstacle detection function needs to be evaluated according to the new standards. 2. **Quantitative risk analysis**: - The paper conducts a system - level hazard rate analysis, showing that after using sensor / perceptron fusion technology, the fusion detection system can meet the acceptable hazard rate applicable to safety integrity level 3 (SIL - 3). - Quantify the hazard rate of the obstacle detection function through parametric stochastic model checking, allowing some values to be undefined for lookup when obtaining reliable values in subsequent experiments. 3. **Mathematical analysis of CNN models**: - The paper proposes a new mathematical analysis technique for identifying the classification clusters and equivalence classes of CNN models, which can be used to determine the residual error probability of the trained CNN and its upper confidence limit. - This gray - box method takes into account the internal structure of the CNN model and is crucial for proving the comprehensive coverage of statistical tests. 4. **Statistical testing methods**: - The paper introduces a new statistical testing strategy for estimating the residual error probability of the trained CNN in the obstacle detection task and providing the corresponding confidence values. - This method is especially suitable for AI system components based on image classification, determining the sample size through Monte Carlo tests and calculating specific estimates. In summary, the paper aims to improve the reliability of CNN - based obstacle detection systems through new verification and confirmation methods, thereby ensuring their safe application in highly automated train control systems.

A Stochastic Approach to Classification Error Estimates in Convolutional Neural Networks

Statistics-Physics-Based Interpretation of the Classification Reliability of Convolutional Neural Networks in Industrial Automation Domain

Probabilistic Risk Assessment of an Obstacle Detection System for GoA 4 Freight Trains

Algorithm-Based Fault Tolerance for Convolutional Neural Networks

Certifying Robustness of Convolutional Neural Networks with Tight Linear Approximation

Non-asymptotic Excess Risk Bounds for Classification with Deep Convolutional Neural Networks

A novel statistical approach to analyze image classification

Reliable Classification with Ensemble Convolutional Neural Networks.

Evaluation and Mitigation of Weight-Related Single Event Upsets in a Convolutional Neural Network

SAFE-OCC: A Novelty Detection Framework for Convolutional Neural Network Sensors and its Application in Process Control

BayesNetCNN: incorporating uncertainty in neural networks for image-based classification tasks

Comprehensive Analysis of Network Robustness Evaluation Based on Convolutional Neural Networks with Spatial Pyramid Pooling

Scalable Surrogate Verification of Image-based Neural Network Control Systems using Composition and Unrolling

Formal Verification of Graph Convolutional Networks with Uncertain Node Features and Uncertain Graph Structure

PaRCE: Probabilistic and Reconstruction-based Competency Estimation for CNN-based Image Classification

Systematic Testing of Convolutional Neural Networks for Autonomous Driving

Verifying Controllers with Convolutional Neural Network-based Perception: A Case for Intelligible, Safe, and Precise Abstractions

Robustness Verification of Classification Deep Neural Networks Via Linear Programming.

Mathematical Modeling and Statistical Exploration of Residual Computing Based Convolutional Neural Network Based Classifier for Complex Image Demarcation

A Rank Decomposed Statistical Error Compensation Technique for Robust Convolutional Neural Networks in the Near Threshold Voltage Regime

Purification Of Contaminated Convolutional Neural Networks Via Robust Recovery: An Approach with Theoretical Guarantee in One-Hidden-Layer Case