Abstract:A fundamental challenge of software testing is the statistically well-grounded extrapolation from program behaviors observed during testing. For instance, a security researcher who has run the fuzzer for a week has currently no means (i) to estimate the total number of feasible program branches, given that only a fraction has been covered so far, (ii) to estimate the additional time required to cover 10% more branches, or (iii) to assess the residual risk that a vulnerability exists when no vulnerability has been discovered. Failing to discover a vulnerability, does not mean that none exists---even if the fuzzer was run for a week (or a year). Hence, testing provides no formal correctness guarantees. In this article, I establish an unexpected connection with the otherwise unrelated scientific field of ecology, and introduce a statistical framework that models Software Testing and Analysis as Discovery of Species (STADS). For instance, in order to study the species diversity of arthropods in a tropical rain forest, ecologists would first sample a large number of individuals from that forest, determine their species, and extrapolate from the properties observed in the sample to properties of the whole forest. The estimation (i) of the total number of species, (ii) of the additional sampling effort required to discover 10% more species, or (iii) of the probability to discover a new species are classical problems in ecology. The STADS framework draws from over three decades of research in ecological biostatistics to address the fundamental extrapolation challenge for automated test generation. Our preliminary empirical study demonstrates a good estimator performance even for a fuzzer with adaptive sampling bias---AFL, a state-of-the-art vulnerability detection tool. The STADS framework provides statistical correctness guarantees with quantifiable accuracy.

Research of Statistical Testing Based on a Usage Model

ANALYZE OF FUNCTIONAL REGRESSION TEST TOOLS

Model Based Load Testing of Web Applications

Combining Model Checking and Testing with an Application to Reliability Prediction and Distribution

LTF: A Model-Based Load Testing Framework for Web Applications

A Goodness-of-Fit Test for Statistical Models

A Path-Oriented Tool Supporting for Testing

A Markov chain model for statistical software testing

Test Suite Reduction via Submodular Function Maximization

Automatic Test Data Generation Based on Usage Model and Genetic Algorithm

StatWhy: Formal Verification Tool for Statistical Hypothesis Testing Programs

Paths Selection for Software Test and Its Performance Analysis Based on Software Usage Model

STADS: Software Testing as Species Discovery

Research on Model of Automated Test Case Generation for Complicated Interactive Software

Measuring Software Testability via Automatically Generated Test Cases

Optimizing design method of operational profile in statistical testing

Study of Regression Testing Tool Based on Subversion

Design and Implement of a Tool for Pairwise Testing Data Generation

Software Reliability Modeling and Cost Estimation Incorporating Testing-Effort and Efficiency

An Approximation-based Approach for the Random Exploration of Large Models

Useful Statistical Methods for Human Factors Research in Software Engineering: A Discussion on Validation with Quantitative Data