Abstract:Due to limited time and resources available for execution, test case selection always remains crucial for cost-effective testing. It is even more prominent when test cases require manual steps, e.g., operating physical equipment. Thus, test case selection must consider complicated trade-offs between cost (e.g., execution time) and effectiveness (e.g., fault detection capability). Based on our industrial collaboration within the Maritime domain, we identified a real-world and multi-objective test case selection problem in the context of robustness testing, where test case execution requires human involvement in certain steps, such as turning on the power supply to a device. The high-level goal is to select test cases for execution within a given time budget, where test engineers provide weights for a set of objectives, depending on testing requirements, standards, and regulations.To address the identified test case selection problem, we defined a fitness function including one cost measure, i.e., Time Difference (TD) and three effectiveness measures, i.e., Mean Priority (MPR), Mean Probability (MPO) and Mean Consequence (MC) that were identified together with test engineers. We further empirically evaluated eight multi-objective search algorithms, which include three weight-based search algorithms (e.g., Alternating Variable Method) and five Pareto-based search algorithms (e.g., Strength Pareto Evolutionary Algorithm 2 (SPEA2)) using two weight assignment strategies (WASs). Notice that Random Search (RS) was used as a comparison baseline. We conducted two sets of empirical evaluations: 1) Using a real world case study that was developed based on our industrial collaboration; 2) Simulating the real world case study to a larger scale to assess the scalability of the search algorithms. Results show that SPEA2 with either of the WASs performed the best for both the studies. Overall, SPEA2 managed to improve on average 32.7%, 39% and 33% in terms of MPR, MPO and MC respectively as compared to RS.

Constructing Cost-Aware Functional Test-Suites Using Nested Differential Evolution Algorithm.

An Experience-Based Approach for Test Execution Effort Estimation

Test Data Generation Algorithm of Combinatorial Testing Based on Differential Evolution

MS-guided Many-Objective Evolutionary Optimisation for Test Suite Minimisation

An Enhanced Adaptive Differential Evolution Algorithm With Multi-Mutation Schemes and Weighted Control Parameter Setting

Test Suite Reduction via Evolutionary Clustering

Evolutionary generation of test suites for multi-path coverage of MPI programs with non-determinism

Search-Based Cost-Effective Test Case Selection Within A Time Budget: an Empirical Study

A Dynamic Optimization Strategy for Evolutionary Testing

TCA: an Efficient Two-Mode Meta-Heuristic Algorithm for Combinatorial Test Generation (T).

Test Selection with Equivalence Class Partitioning

A Study of Enhanced MC/DC Coverage Criterion for Software Testing.

Differential evolution algorithm with dichotomy-based parameter space compression

Enhance Combinatorial Testing with Metamorphic Relations

Exploring Multiple-Objective Optimization for Efficient and Effective Test Paper Design with Dynamic Programming Guided Genetic Algorithm

A General Enhancement Method for Test Strategy Generation for the Sequential Fault Diagnosis of Complex Systems

A Generic Test Suite for Evolutionary Multifidelity Optimization

Configuration Strategies for Evolutionary Testing.

Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization

Comparative Analysis of Constraint Handling Techniques for Constrained Combinatorial Testing.

An Effective Approach to High Strength Covering Array Generation in Combinatorial Testing