Abstract:We show that there exists a trade-off between high rate of combination coverage and low switching cost.We comprehensively compare three single objective algorithms with hybrid and multi-objective algorithms in the presence of switching cost.We find that the heuristic solver for travelling salesmen problem is the best algorithm to minimize switching cost.In order to improve the rate of combination coverage over time and the ability of early fault detection, we recommend multi-objective algorithms. Display Omitted ContextTest suite prioritization is a problem of deciding the order of executing test cases to reach the desirable outcome. Many cost-cognisant prioritization approaches decide the order based on the cost of test execution; but few based on the cost of switching test cases. The latter known as switching cost is the effort of re-configuring the environment for running subsequent test cases. Our previous studies show that switching cost can affect the efficiency of testing. ObjectiveIn this paper, we aim to identify the optimal testing order that can detect interaction triggered faults earlier in the presence switching cost. MethodWe presented a distance based metric to measure the switching cost between test cases. As reducing the switching cost can make the whole test suite run faster and thus achieve full combination coverage earlier, single-objective algorithms were used to minimize the total switching cost. Besides, when determining the next test case to run, there is a trade-off between high combination coverage and low switching cost. Hence, hybrid and multi-objective algorithms were used to achieve a better balance. In order to evaluate different algorithms, we conducted a series of experiments covering 400 different testing scenarios. We also conducted an empirical study with six real world applications. ResultsThe heuristic solver for the travelling salesman problem is the best algorithm to minimize the switching cost. It can detect faults earlier than the order with high rate of combination coverage. But in order to further reduce the effort to detect the first fault, the hybrid and multi-objective algorithms are the best methods. ConclusionPrioritization based on switching cost can speed up the fault detection to some extent, but prioritization based on both combination coverage and testing cost can deliver the optimal testing order.

Test Sequencing Strategy with Imperfect Test

Improvement of reassignment strategy in defects removal based on queuing theory and simulation

Simulation of the defect removal process with queuing theory

A test sequences optimization method for improving fault coverage

Study of protocol conformance test sequence generation algorithm based on FSM

A Stochastic Combinatorial Optimization Model for Test Sequence Optimization.

Reducing Test Sequence Length Using Invertible Sequences

A Parallel Implementation Strategy of Adaptive Testing

PRIORITIZATION OF COMBINATORIAL TEST CASES BY INCREMENTAL INTERACTION COVERAGE

Test Pattern Merging Algorithms for Sequential Circuits

Incremental Sequence Pair Generation Algorithm for SOC Test Scheduling

The optimal testing order in the presence of switching cost.

Test Case Prioritization Incorporating Ordered Sequence of Program Elements

An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction Identification

Directed test suite augmentation via exploiting program dependency.

Generating External Synchronizable Test Sequences That Detect Output-Shifting Faults

A hybrid approach to build prioritized pairwise interaction test suites

JUnit test case simplification method based on execute sequences

Adaptive Interaction Fault Location Based on Combinatorial Testing

Defect-Oriented Test Case Prioritization

Top-down: A better strategy for incremental covering array generation