Special Issue on Mutation Testing and Analysis
René Just,Jens Krinke,Nan Li,José Miguel Rojas
DOI: https://doi.org/10.1002/stvr.1697
2019-01-01
Abstract:Program mutation is acknowledged as an important method to assess the fault-finding capabilities of an existing test suite (mutation analysis) and to generate efficient and effective test suites (mutation testing). Mutation has mostly been applied at the source code level, but more recently, related ideas have also been used to test artefacts described in a considerable variety of notations and at different levels of abstraction. Ideas derived from mutation testing are used with requirements, formal specifications, models, architectural design notations, and informal descriptions. Data mutation has also been investigated for critical systems, embedded systems, and web services. Mutation is thus an established method in verification and validation of software and hardware systems. This special issue covers a wide variety of research areas connected to program mutation, mutation analysis, and mutation testing. Specifically, it covers theoretical analyses over mutant subsumption, cost reduction with mutant clustering, new mutation operators for feature-oriented software product lines, and applications of mutation analysis to prioritize tests, evaluate model clone detectors, and generate tests for regular expressions. This special issue includes three extended manuscripts, previously presented at the International Workshop on Mutation Analysis in years 2016 and 2017. Since 2009, the mutation analysis workshop has been an annual event where researchers and practitioners meet to discuss new and emerging trends in mutation analysis, and so far, there have been 14 total occurrences of the workshop. The guest editors would like to thank all the authors of this special issue for their contributions and the reviewers for devoting their time. We hope the reader will enjoy this special issue. Mutation testing suffers from redundant mutants, that is, mutants that do not contribute to the overall quality of the test suite since any test suite that detects all the non-redundant mutants is guaranteed to detect all the redundant mutants as well. Redundant mutants might cause problems when benchmarking testing techniques since they can get high scores without detecting any non-redundant mutants. This paper addresses the problem of subsuming mutants, a class of redundant mutants, and focuses on mutants in logical expressions. Besides a thorough investigation of such mutants, the authors show that the probability for a random test to experience the problem can be more than 30% and that 50% of the mutants might be affected in a real software system. This paper examines the possibility of reducing the cost of statement-level mutant clustering by comparing the number of mutant executions with those of expression-level and block-level mutant clustering. The experiments were conducted with nine real-world programs, and the results show that statement-level clustering is more cost-effective than block-level clustering when considering their clustering overheads. The authors also discover a compound expression that plays a major role in providing a cost-reduction effect in statement-level clustering. With the compound expression, the number of candidate mutants to be clustered in a statement scope increases, and state change can be comprehensively examined, thereby increasing the possibility of cost reduction. The authors propose a new test case prioritization technique that combines both mutation-based and diversity-aware approaches. This diversity-aware mutation-based technique relies on the notion of mutant distinguishment, which distinguishes one mutant's behaviour from another, rather than from the original program. The relative cost and effectiveness of the mutation-based prioritization techniques (i.e. using both the traditional mutant kill and the proposed mutant distinguishment) are empirically investigated with 352 real faults and 553 477 developer-written test cases. The authors find that no single technique dominates across all the studied faults and analyses the reasons. The development of increasingly complex software systems requires means to abstract away some of this complexity. Model-based or model-driven software development approaches allow software engineers to reason about software in more abstract terms where modelling languages rather than programming languages are used to develop software systems. As a result, there is a growing need to support maintainability and evolvability for such model-based development processes. Redundancy between models, referred to as model clones, is one of the problems that may arise in model-based software development. While several techniques and tools have been proposed to detect such model clones, assessing their effectiveness is an open problem. This paper addresses this problem and proposes a novel evaluation framework named MuMonDE that compares model clone detectors using mutation analysis. The framework applies mutations at the model level, and the resulting mutants, which emulate different types of model clones, are used to evaluate the performance of existing clone detectors. An empirical study comparing the ConQAT and SIMONE model clone detectors for Simulink demonstrates the effectiveness of the MuMonDE framework. Regular expressions are frequently used in software systems, for example, for input or data validation. However, writing a regular expression that precisely captures the desired set of strings is difficult and error prone. This paper proposes a fault-based approach for generating strings that are usable as tests for regular expressions. First, it defines classes of representative faults and introduces the notion of distinguishing strings (strings that are able to expose a given fault). Second, it proposes an approach that generates a test suite, composed of distinguishing strings, that is able to reveal potential faults in a given regular expression. Third, it evaluates the proposed approach and provides a comparison with other test generation tools. Software product lines (SPL) is a mature, trending software engineering approach. Testing SPL systems, and in particular performing mutation testing on them, remains an open problem due to the multiple challenges unique to this technology. This paper proposes six mutation operators for composition-based product lines, which resemble variability faults, that is, faults caused by combinations of features that exist only on some of the products supported by the product line. The effectiveness and usefulness of the proposed operators is evaluated against classical mutation operators on two case studies. The empirical evaluation indicates that the proposed operators are suitable to cause variability faults and therefore extend the capabilities of conventional mutation operators.