Mutation Testing in Evolving Systems: Studying the relevance of mutants to code evolution

Milos Ojdanic,Ezekiel Soremekun,Renzo Degiovanni,Mike Papadakis,Yves Le Traon
DOI: https://doi.org/10.48550/arXiv.2112.14566
2021-12-29
Abstract:When software evolves, opportunities for introducing faults appear. Therefore, it is important to test the evolved program behaviors during each evolution cycle. We conduct an exploratory study to investigate the properties of commit-relevant mutants, i.e., the test elements of commit-aware mutation testing, by offering a general definition and an experimental approach to identify them. We thus, aim at investigating the prevalence, location, comparative advantages of commit-aware mutation testing over time (i.e., the program evolution) and the predictive power of several commit-related features to understand the essential properties for its best-effort application case. Our approach utilizes the impact of mutants and the effects of one mutant on another in capturing and analyzing the implicit interactions between the changed and unchanged code parts. The study analyses millions of mutants (over 10 million), 288 commits, five (5) different open-source software projects involving over 68,213 CPU days of computation and sets a ground truth where we perform our analysis. Our analysis shows that commit-relevant mutants are located mainly outside of program commit change (81%), while an effective selection of commit-relevant mutants can reduce the number of mutants by up to 93%. In addition, we demonstrate that commit relevant mutation testing is significantly more effective and efficient than state-of-the-art baselines. Our analysis of the predictive power of mutants and commit-related features in predicting commit-relevant mutants found that most proxy features do not reliably predict commit-relevant mutants. This empirical study highlights the properties of commit-relevant mutants and demonstrates the importance of identifying and selecting commit-relevant mutants when testing evolving software systems.
Software Engineering
What problem does this paper attempt to address?
The problem that this paper attempts to solve is how to effectively conduct mutation testing during the evolution of software systems. Specifically, the paper focuses on how to identify and select commit - relevant mutants after software code changes, in order to improve the effectiveness and efficiency of testing. Traditional mutation testing methods usually inject mutants into the entire code base, which becomes impractical when dealing with large or frequently changing software projects. Therefore, the paper proposes a method called "commit - aware mutation testing", aiming to optimize the mutation testing process by focusing on the changed parts of the program. ### Main research objectives of the paper: 1. **Explore the properties of commit - relevant mutants**: The paper identifies these mutants through definition and experimental methods, and studies their distribution, location, universality, predictability, and practicality. 2. **Evaluate the advantages of commit - aware mutation testing**: Compare this method with traditional methods of randomly selecting mutants or only analyzing mutants in the changed parts, and evaluate its advantages in reducing the number of test executions and improving test efficiency. 3. **Explore the characteristics for predicting commit - relevant mutants**: Study whether some characteristics related to commits (such as commit size, mutation type, etc.) can reliably predict commit - relevant mutants. ### Main findings: - **Commit - relevant mutants are widespread**: In the evaluation, on average 30% of the mutants are commit - relevant. - **Selecting subsuming commit - relevant mutants significantly reduces the number of mutants**: Selecting such mutants can further reduce the number of mutants, with an average reduction of about 93%. - **A large number of commit - relevant mutants are located outside the program changes**: Most commit - relevant mutants (69%) are located outside the unchanged methods. - **Existing mutant selection methods cannot effectively predict commit - relevant mutants**: For example, characteristics such as commit size or mutation operator type cannot reliably predict commit - relevant mutants. - **Commit - aware mutation testing significantly reduces the number of test executions**: Compared with the method of randomly selecting mutants, commit - aware mutation testing can reduce the number of test executions by about 16 times. ### Methodology: - **Define commit - relevant mutants**: Based on the concept of observational slicing, identify these mutants through higher - order mutants. - **Experimental design**: Analyzed more than 10 million mutants and 288 commits, involving 5 mature open - source software projects, and the experiment took more than 68,213 CPU days. ### Conclusion: Through extensive empirical research, the paper reveals the properties of commit - relevant mutants and proves the importance and effectiveness of commit - aware mutation testing in testing evolving systems. This method can not only significantly reduce the testing cost, but also more effectively identify and test the program behaviors affected by changes.