Abstract:Background. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive. Therefore, many mutation reduction strategies, which aim to reduce the number of mutants, have been proposed. Problem. It is important to measure the ability of a mutation reduction strategy to maintain test suite effectiveness evaluation. However, existing evaluation indicators are unable to measure the “order-preserving ability”, i.e., to what extent the mutation score order among test suites is maintained before and after mutation reduction. As a result, misleading conclusions can be achieved when using existing indicators to evaluate the reduction effectiveness. Objective. We aim to propose evaluation indicators to measure the “order-preserving ability” of a mutation reduction strategy, which is important but missing in our community. Method. Given a test suite on a Software Under Test (SUT) with a set of original mutants, we leverage the test suite to generate a group of test suites that have a partial order relationship in defect detecting ability. When evaluating a reduction strategy, we first construct two partial order relationships among the generated test suites in terms of mutation score, one with the original mutants and another with the reduced mutants. Then, we measure the extent to which the partial order under the original mutants remains unchanged in the partial order under the reduced mutants. The more partial order is unchanged, the stronger the Order Preservation ( OP ) of the mutation reduction strategy is, and the more effective the reduction strategy is. Furthermore, we propose Effort-aware Relative Order Preservation ( EROP ) to measure how much gain a mutation reduction strategy can provide compared with a random reduction strategy. Result. The experimental results show that OP and EROP are able to efficiently measure the “order-preserving ability” of a mutation reduction strategy. As a result, they have a better ability to distinguish various mutation reduction strategies compared with the existing evaluation indicators. In addition, we find that Subsuming Mutant Selection (SMS) and Clustering Mutant Selection (CMS) are more effective than the other strategies under OP and EROP. Conclusion. We suggest, for the researchers, that OP and EROP should be used to measure the effectiveness of a mutant reduction strategy, and for the practitioners, that SMS and CMS should be given priority in practice.

Second-Order Mutation Testing Cost Reduction Based on Mutant Clustering using SOM Neural Network Model

A Cost-effective and Machine-learning-based method to identify and cluster redundant mutants in software mutation testing

How Higher Order Mutant Testing Performs for Deep Learning Models: A Fine-Grained Evaluation of Test Effectiveness and Efficiency Improved from Second-Order Mutant-Classification Tuples

Efficient software mutation test by clustering the single-line redundant mutants

A Novel Method of Mutation Clustering Based on Domain Analysis

Test Data Generation for Mutation Testing Based on Markov Chain Usage Model and Estimation of Distribution Algorithm

An Approach to Test Data Generation for Killing Multiple Mutants

Faster Mutation Analysis with Fewer Processes and Smaller Overheads

MQP: Mutants Quality Prediction for Cost-Effective Mutation Testing

Spotting Code Mutation for Predictive Mutation Testing

Mutation Operator Reduction for Cost-effective Deep Learning Software Testing Via Decision Boundary Change Measurement

Mutation Operator Reduction for Deep Learning System

DeepKernel: 2D-kernels clustering based mutant reduction for cost-effective deep learning model testing

HMER: A Hybrid Mutation Execution Reduction approach for Mutation-based Fault Localization

Mutant reduction evaluation: what is there and what is missing?

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

Large Language Models for Equivalent Mutant Detection: How Far Are We?

Can Higher-Order Mutants Improve the Performance of Mutation-Based Fault Localization?

MUPPAAL: Efficient Elimination and Reduction of Useless Mutants in Real‐Time Model‐Based Systems

Predictive Mutation Testing

Heuristics for Strongly Killing Mutants in Software Mutation Testing