Abstract:Knowing the effectiveness of a test suite is essential for many activities such as assessing the test adequacy of code and guiding the generation of new test cases. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive, as a large number of mutants (buggy versions) are needed to be generated from a production code under test and executed against the test suite. In order to reduce the expensive testing cost, recent studies proposed to use supervised models to predict the effectiveness of a test suite without executing the test suite against the mutants. Nonetheless, the training of such a supervised model requires labeled data, which still depends on the costly mutant execution. Furthermore, existing models are based on traditional supervised learning techniques, which assume that the training and testing data come from the same distribution. But, in practice, software systems are subject to considerable concept drifts, i.e., the same distribution assumption usually does not hold. This can lead to inaccurate predictions of a learned supervised model on the target code as time progresses. To tackle these problems, in this paper, we propose a Coverage-Based Unsupervised Approach (CBUA) for evaluating the effectiveness of a test suite. Given a production code under test, the corresponding mutants, and a test suite, CBUA first collects the coverage information of the mutated statements in the target production code under the execution of the test suite. Then, CBUA employs coverage to estimate the probability of each mutant being alive. As such, a mutation score is computed to evaluate the test suite effectiveness and the predicted labels (i.e., killed or alive) are obtained. The whole process only requires a one-time execution of the test suite against the target production code, without involving any mutant execution and any training data. CBUA can ensure the score monotonicity property (i.e., adding test cases to a test suite does not decrease its mutation score), which may be violated by a supervised approach. The experimental results show that CBUA is very competitive with the state-of-the-art supervised approaches in prediction accuracy. In particular, CBUA is shown to be more effective in finding mutants that are covered but not killed by a test suite, which is helpful in identifying the weaknesses in the current test suite and generating new test cases accordingly. Since CBUA is an easy-to-implement approach with a low cost, we suggest that it should be used as a baseline approach for comparison when any novel prediction approach is proposed in future studies.

Test Case Level Predictive Mutation Testing Combining PIE and Natural Language Features

Predictive Mutation Testing

A Model for Predicting Statement Mutation Scores

A Method for Finding Redundant Mutants in Mutation Testing

Spotting Code Mutation for Predictive Mutation Testing

MQP: Mutants Quality Prediction for Cost-Effective Mutation Testing

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

Scalable Mutation Testing Using Predictive Analysis of Deep Learning Model

An Approach to Test Data Generation for Killing Multiple Mutants

Hybrid mutation driven testing for natural language inference

CBUA: A Probabilistic, Predictive, and Practical Approach for Evaluating Test Suite Effectiveness

Test Case Generation Based on Mutation Analysis and Set Evolution

Enhancement of Mutation Testing Via Fuzzy Clustering and Multi-Population Genetic Algorithm

Effective test generation using pre-trained Large Language Models and mutation testing

Predictive Mutation-Based Fault Localization: Balancing Effectiveness and Cost

Efficient Mutation Testing via Pre-Trained Language Models

Test Case Prioritization and Mutation Testing.

An Exploratory Study on Using Large Language Models for Mutation Testing

Test Oracle Prediction for Mutation Based Fault Localization

A novel method for mutation based program testing