Abstract:Classification neural networks fail to detect inputs that do not fall inside the classes they have been trained for. Runtime monitoring techniques on the neuron activation pattern can be used to detect such inputs. We present an approach for monitoring classification systems via data abstraction. Data abstraction relies on the notion of box with a resolution. Box-based abstraction consists in representing a set of values by its minimal and maximal values in each dimension. We augment boxes with a notion of resolution and define their clustering coverage, which is intuitively a quantitative metric that indicates the abstraction quality. This allows studying the effect of different clustering parameters on the constructed boxes and estimating an interval of sub-optimal parameters. Moreover, we automatically construct monitors that leverage both the correct and incorrect behaviors of a system. This allows checking the size of the monitor abstractions and analyzing the separability of the network. Monitors are obtained by combining the sub-monitors of each class of the system placed at some selected layers. Our experiments demonstrate the effectiveness of our clustering coverage estimation and show how to assess the effectiveness and precision of monitors according to the selected clustering parameter and monitored layers.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to detect inputs that are not within the scope of training categories in a neural network classification system. Specifically, the article proposes a customizable reference runtime monitoring method based on resolution boxes for runtime verification of neural networks. This method represents a set of values by abstracting data and introduces the concept of resolution, thereby defining the quantitative metric of clustering coverage. This helps to study the influence of different clustering parameters on the constructed boxes and estimate sub - optimal parameter intervals. ### Core problems of the paper 1. **Detecting unseen inputs**: Classification neural networks are unable to detect inputs that do not belong to their training categories. 2. **Runtime monitoring techniques**: Runtime monitoring techniques using neuron activation patterns can be used to detect these inputs. 3. **Data abstraction and resolution boxes**: Introduce box abstraction with resolution to represent and monitor the behavior of neural networks more precisely. 4. **Automatically constructing monitors**: Use correct and incorrect behaviors as references to automatically construct monitors to check the abstract size of the network and analyze its separability. ### Main contributions - **Introducing uncertainty judgment**: When the positive and negative sample regions of the abstraction overlap, mark the input as "uncertain" and provide feedback to the network designer. - **Clustering coverage**: Evaluate the precision of boxes by measuring clustering coverage, and balance precision and computational overhead. - **Optimizing clustering parameter selection**: Improve the precision and recall rate of monitors by adjusting clustering parameters. ### Method overview 1. **Data abstraction**: Abstract a high - dimensional vector set into boxes with resolution. 2. **Clustering algorithm**: Apply clustering algorithms such as k - means to divide the data into multiple clusters. 3. **Box - abstraction construction**: Construct box - abstractions for each cluster and calculate the clustering coverage. 4. **Monitor construction**: Combine the abstractions of correct and incorrect behaviors to construct a runtime monitor. ### Experimental results Experiments show that this method can effectively evaluate clustering coverage and evaluate the effectiveness and precision of monitors according to the selected clustering parameters and monitoring layers. ### Formula summary - **Box - abstraction**: \[ B(X)=\{(x_1,\dots,x_n)\in\mathbb{R}^n\mid\bigwedge_{i = 1}^n a_i\leq x_i\leq b_i\} \] where \(a_i=\min(\{\theta_i(x)\})\) and \(b_i=\max(\{\theta_i(x)\})\). - **Number of covered cells**: \[ |CovCell(b)|=\prod_{i = 1}^n n_i,\quad\text{where}\quad n_i = \begin{cases} \left\lceil|X|\times\frac{a'_i - a_i}{b_i - a_i}\right\rceil-\left\lceil|X|\times\frac{b'_i - a_i}{b_i - a_i}\right\rceil + 1&\text{if }b'_i\neq a'_i\\ 1&\text{otherwise} \end{cases} \] - **Sub - box coverage**: \[ coverage_X(b)=\frac{|CovCell(b)|}{|X|^{|CovB(X)|}} \] - **Clustering coverage estimation**: \[ r_l=r_u-\sum_{b\in B_{int}}coverage_X(b),\quad r_u=\sum_{b\in BX}coverage_X(b) \] Through the above methods, this paper aims to improve the precision and reliability of neural network runtime monitoring, especially when dealing with unseen inputs.

Customizable Reference Runtime Monitoring of Neural Networks using Resolution Boxes

Into the Unknown: Active Monitoring of Neural Networks

A Wafer Surface Defect Detection Method Built on Generic Object Detection Network

Runtime Monitoring Neuron Activation Patterns

Cognitive Sensing: Adaptive Anomalies Detection with Deep Networks

Monitizer: Automating Design and Evaluation of Neural Network Monitors

A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networks

Can we Defend Against the Unknown? An Empirical Study About Threshold Selection for Neural Network Monitoring

Runtime Monitoring DNN-Based Perception

Provably-Robust Runtime Monitoring of Neuron Activation Patterns

Opening the Black Box of Neural Networks: Methods for Interpreting Neural Network Models in Clinical Applications

Benchmarking Safety Monitors for Image Classifiers with Machine Learning

Hack The Box: Fooling Deep Learning Abstraction-Based Monitors

Small object detection using retinanet with hybrid anchor box hyper tuning using interface of Bayesian mathematics

Out-Of-Distribution Detection Is Not All You Need

Collaborative Inference for Efficient Remote Monitoring

Towards a Flexible and High-Fidelity Approach to Distributed DNN Training Emulation

Optimal Classification-based Anomaly Detection with Neural Networks: Theory and Practice

From Black-box to White-box: Examining Confidence Calibration under different Conditions

A Neural Network Anomaly Detector Using the Random Cluster Model

An Effective Multi-Resolution Hierarchical Granular Representation based Classifier using General Fuzzy Min-Max Neural Network