Abstract:We show that there exists a trade-off between high rate of combination coverage and low switching cost.We comprehensively compare three single objective algorithms with hybrid and multi-objective algorithms in the presence of switching cost.We find that the heuristic solver for travelling salesmen problem is the best algorithm to minimize switching cost.In order to improve the rate of combination coverage over time and the ability of early fault detection, we recommend multi-objective algorithms. Display Omitted ContextTest suite prioritization is a problem of deciding the order of executing test cases to reach the desirable outcome. Many cost-cognisant prioritization approaches decide the order based on the cost of test execution; but few based on the cost of switching test cases. The latter known as switching cost is the effort of re-configuring the environment for running subsequent test cases. Our previous studies show that switching cost can affect the efficiency of testing. ObjectiveIn this paper, we aim to identify the optimal testing order that can detect interaction triggered faults earlier in the presence switching cost. MethodWe presented a distance based metric to measure the switching cost between test cases. As reducing the switching cost can make the whole test suite run faster and thus achieve full combination coverage earlier, single-objective algorithms were used to minimize the total switching cost. Besides, when determining the next test case to run, there is a trade-off between high combination coverage and low switching cost. Hence, hybrid and multi-objective algorithms were used to achieve a better balance. In order to evaluate different algorithms, we conducted a series of experiments covering 400 different testing scenarios. We also conducted an empirical study with six real world applications. ResultsThe heuristic solver for the travelling salesman problem is the best algorithm to minimize the switching cost. It can detect faults earlier than the order with high rate of combination coverage. But in order to further reduce the effort to detect the first fault, the hybrid and multi-objective algorithms are the best methods. ConclusionPrioritization based on switching cost can speed up the fault detection to some extent, but prioritization based on both combination coverage and testing cost can deliver the optimal testing order.

A Stochastic Combinatorial Optimization Model for Test Sequence Optimization.

An Novel Testing Sequence Optimization Method under Dynamic Environments

An Improved Test Selection Optimization Model Based on Fault Ambiguity Group Isolation and Chaotic Discrete PSO.

Towards Optimal Concolic Testing

A test sequences optimization method for improving fault coverage

Methods of Sequential Test Optimization in Dynamic Environment

A General Enhancement Method for Test Strategy Generation for the Sequential Fault Diagnosis of Complex Systems

Generating Variable Strength Combinatorial Test Suite with One-test-at-a-time Strategy

PRIORITIZATION OF COMBINATORIAL TEST CASES BY INCREMENTAL INTERACTION COVERAGE

A Novel Test Optimizing Algorithm for Sequential Fault Diagnosis.

On Test Sequence Generation using Multi-Objective Particle Swarm Optimization

Petri Net Sequence Optimization Algorithm Based on Test Importance

The optimal testing order in the presence of switching cost.

Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite.

Dynamic Testing Method Based on Test Sequence Fail Tree

Employment of Multiple Algorithms for Optimal Path-based Test Selection Strategy

Functional Test-Cost Reduction Based on Fault Tree Analysis and Binary Optimization

A Markov Decision Approach to Optimize Testing Profile in Software Testing

Stochastic Systems Simulation Optimization

Optimal Testing Strategies in Overlapped Design Process.

A comparison of sample-path-based simulation-optimization and stochastic decomposition for multi-location transshipment problems