Abstract:With the introduction of neuron coverage as a testing criterion for deep neural networks (DNNs), covering more neurons to detect more internal logic of DNNs became the main goal of many research studies. While some works had made progress, some new challenges for testing methods based on neuron coverage had been proposed, mainly as establishing better neuron selection and activation strategies influenced not only obtaining higher neuron coverage, but also more testing efficiency, validating testing results automatically, labeling generated test cases to extricate manual work, and so on. In this article, we put forward Test4Deep, an effective white-box testing DNN approach based on neuron coverage. It is based on a differential testing framework to automatically verify inconsistent DNNs' behavior. We designed a strategy that can track inactive neurons and constantly triggered them in each iteration to maximize neuron coverage. Furthermore, we devised an optimization function that guided the DNN under testing to deviate predictions between the original input and generated test data and dominated unobservable generation perturbations to avoid manually checking test oracles. We conducted comparative experiments with two state-of-the-art white-box testing methods DLFuzz and DeepXplore. Empirical results on three popular datasets with nine DNNs demonstrated that compared to DLFuzz and DeepXplore, Test4Deep, on average, exceeded by 32.87% and 35.69% in neuron coverage, while reducing 58.37% and 53.24% testing time, respectively. In the meantime, Test4Deep also produced 58.37% and 53.24% more test cases with 23.81% and 98.40% fewer perturbations. Even compared with the two highest neuron coverage strategies of DLFuzz, Test4Deep still enhanced neuron coverage by 4.34% and 23.23% and achieved 94.48% and 85.67% higher generation time efficiency. Furthermore, Test4Deep could improve the accuracy and robustness of DNNs by merging generated test cases and retraining.

DeepSI: A Sensitive-Driven Testing Samples Generation Method of Whitebox CNN Model for Edge Computing

White-box Fairness Testing Through Adversarial Sampling

There is Limited Correlation Between Coverage and Robustness for Deep Neural Networks

When Single Event Upset Meets Deep Neural Networks: Observations, Explorations, and Remedies

Intrusion Detection for Secure Social Internet of Things Based on Collaborative Edge Computing: A Generative Adversarial Network-Based Approach.

Excitement Surfeited Turns to Errors: Deep Learning Testing Framework Based on Excitable Neurons

DeepCNP: an Efficient White-Box Testing of Deep Neural Networks by Aligning Critical Neuron Paths

DeepIA: An Interpretability Analysis based Test Data Generation Method for DNN

DeepXplore: Automated Whitebox Testing of Deep Learning Systems

A White-Box Testing for Deep Neural Networks Based on Neuron Coverage.

Combinatorial Testing for Deep Learning Systems

Neuron Sensitivity Guided Test Case Selection for Deep Learning Testing

VerIDeep: Verifying Integrity of Deep Neural Networks through Sensitive-Sample Fingerprinting

Toward Efficiently Evaluating the Robustness of Deep Neural Networks in IoT Systems: A GAN-Based Method

Intelligent Intrusion Detection for Internet of Things Security: A Deep Convolutional Generative Adversarial Network-enabled Approach

Physical security of deep learning on edge devices: Comprehensive evaluation of fault injection attack vectors

TSDTest: A Efficient Coverage Guided Two-Stage Testing for Deep Learning Systems

Dct: Differential Combination Testing Of Deep Learning Systems

A Novel Supervised Deep Learning Solution to Detect Distributed Denial of Service (DDoS) attacks on Edge Systems using Convolutional Neural Networks (CNN)

DeepCT: Tomographic Combinatorial Testing for Deep Learning Systems

A Cross-Layer Fault Propagation Analysis Method for Edge Intelligence Systems Deployed with DNNs