Abstract:Knowing the effectiveness of a test suite is essential for many activities such as assessing the test adequacy of code and guiding the generation of new test cases. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive, as a large number of mutants (buggy versions) are needed to be generated from a production code under test and executed against the test suite. In order to reduce the expensive testing cost, recent studies proposed to use supervised models to predict the effectiveness of a test suite without executing the test suite against the mutants. Nonetheless, the training of such a supervised model requires labeled data, which still depends on the costly mutant execution. Furthermore, existing models are based on traditional supervised learning techniques, which assume that the training and testing data come from the same distribution. But, in practice, software systems are subject to considerable concept drifts, i.e., the same distribution assumption usually does not hold. This can lead to inaccurate predictions of a learned supervised model on the target code as time progresses. To tackle these problems, in this paper, we propose a Coverage-Based Unsupervised Approach (CBUA) for evaluating the effectiveness of a test suite. Given a production code under test, the corresponding mutants, and a test suite, CBUA first collects the coverage information of the mutated statements in the target production code under the execution of the test suite. Then, CBUA employs coverage to estimate the probability of each mutant being alive. As such, a mutation score is computed to evaluate the test suite effectiveness and the predicted labels (i.e., killed or alive) are obtained. The whole process only requires a one-time execution of the test suite against the target production code, without involving any mutant execution and any training data. CBUA can ensure the score monotonicity property (i.e., adding test cases to a test suite does not decrease its mutation score), which may be violated by a supervised approach. The experimental results show that CBUA is very competitive with the state-of-the-art supervised approaches in prediction accuracy. In particular, CBUA is shown to be more effective in finding mutants that are covered but not killed by a test suite, which is helpful in identifying the weaknesses in the current test suite and generating new test cases accordingly. Since CBUA is an easy-to-implement approach with a low cost, we suggest that it should be used as a baseline approach for comparison when any novel prediction approach is proposed in future studies.

Mutation-driven Test Case Generation Using Short-lived Concurrent Mutants -- First Results

MUT: Human-in-the-Loop Unit Test Migration

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

Effective test generation using pre-trained Large Language Models and mutation testing

An Approach to Test Data Generation for Killing Multiple Mutants

Automatic Test Generation for Mutation Testing on Database Applications.

Test Data Generation for Mutation Testing Based on Markov Chain Usage Model and Estimation of Distribution Algorithm

MMT: Mutation Testing of Java Bytecode with Model Transformation -- An Illustrative Demonstration

MUPPAAL: Efficient Elimination and Reduction of Useless Mutants in Real‐Time Model‐Based Systems

Mutation-based Test Generation for Quantum Programs with Multi-Objective Search

Positive and Negative Testing with Mutation-Driven Model Checking

Predictive Mutation Testing

Test Case Generation using Mutation Operators and Fault Classification

Efficient Mutation Testing via Pre-Trained Language Models

Test generation via Dynamic Symbolic Execution for mutation testing

An Exploratory Study on Using Large Language Models for Mutation Testing

LLMorpheus: Mutation Testing using Large Language Models

Efficient Mutation-Analysis Coverage For Constrained Random Verification

Random unit-test generation with MUT-aware sequence recommendation.

CBUA: A Probabilistic, Predictive, and Practical Approach for Evaluating Test Suite Effectiveness