Abstract:Background. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive. Therefore, many mutation reduction strategies, which aim to reduce the number of mutants, have been proposed. Problem. It is important to measure the ability of a mutation reduction strategy to maintain test suite effectiveness evaluation. However, existing evaluation indicators are unable to measure the “order-preserving ability”, i.e., to what extent the mutation score order among test suites is maintained before and after mutation reduction. As a result, misleading conclusions can be achieved when using existing indicators to evaluate the reduction effectiveness. Objective. We aim to propose evaluation indicators to measure the “order-preserving ability” of a mutation reduction strategy, which is important but missing in our community. Method. Given a test suite on a Software Under Test (SUT) with a set of original mutants, we leverage the test suite to generate a group of test suites that have a partial order relationship in defect detecting ability. When evaluating a reduction strategy, we first construct two partial order relationships among the generated test suites in terms of mutation score, one with the original mutants and another with the reduced mutants. Then, we measure the extent to which the partial order under the original mutants remains unchanged in the partial order under the reduced mutants. The more partial order is unchanged, the stronger the Order Preservation ( OP ) of the mutation reduction strategy is, and the more effective the reduction strategy is. Furthermore, we propose Effort-aware Relative Order Preservation ( EROP ) to measure how much gain a mutation reduction strategy can provide compared with a random reduction strategy. Result. The experimental results show that OP and EROP are able to efficiently measure the “order-preserving ability” of a mutation reduction strategy. As a result, they have a better ability to distinguish various mutation reduction strategies compared with the existing evaluation indicators. In addition, we find that Subsuming Mutant Selection (SMS) and Clustering Mutant Selection (CMS) are more effective than the other strategies under OP and EROP. Conclusion. We suggest, for the researchers, that OP and EROP should be used to measure the effectiveness of a mutant reduction strategy, and for the practitioners, that SMS and CMS should be given priority in practice.

Subsuming Mutation Operators

Mutator-driven Continuous Genome Evolution of Saccharomyces Cerevisiae

Mutation Testing of Memory-Related Operators

Is Operator-Based Mutant Selection Superior to Random Mutant Selection?

Dynamic Mutant Subsumption Analysis using LittleDarwin

Mutant reduction evaluation: what is there and what is missing?

MuRS: Mutant Ranking and Suppression using Identifier Templates

Mopt: Optimized Mutation Scheduling For Fuzzers

Mutation Operator Reduction for Deep Learning System

Automatic Debugging of Operator Errors Based on Efficient Mutation Analysis

Source Code Optimization using Equivalent Mutants

How Higher Order Mutant Testing Performs for Deep Learning Models: A Fine-Grained Evaluation of Test Effectiveness and Efficiency Improved from Second-Order Mutant-Classification Tuples

Mutation Operator Reduction for Cost-effective Deep Learning Software Testing Via Decision Boundary Change Measurement

SuMo: A Mutation Testing Strategy for Solidity Smart Contracts

Retrieval-Enhanced Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model

MMOS: Multi-Staged Mutation Operator Scheduling for Deep Learning Library Testing

Spotting Code Mutation for Predictive Mutation Testing

Large Language Models for Equivalent Mutant Detection: How Far Are We?

MutaPLM: Protein Language Modeling for Mutation Explanation and Engineering

An Exploratory Study on Using Large Language Models for Mutation Testing

Vulnerability Mimicking Mutants