The optimal testing order in the presence of switching cost.

Huayao Wu,Changhai Nie,Fei-Ching Kuo

DOI: https://doi.org/10.1016/j.infsof.2016.08.006

2016-01-01

Abstract:We show that there exists a trade-off between high rate of combination coverage and low switching cost.We comprehensively compare three single objective algorithms with hybrid and multi-objective algorithms in the presence of switching cost.We find that the heuristic solver for travelling salesmen problem is the best algorithm to minimize switching cost.In order to improve the rate of combination coverage over time and the ability of early fault detection, we recommend multi-objective algorithms. Display Omitted ContextTest suite prioritization is a problem of deciding the order of executing test cases to reach the desirable outcome. Many cost-cognisant prioritization approaches decide the order based on the cost of test execution; but few based on the cost of switching test cases. The latter known as switching cost is the effort of re-configuring the environment for running subsequent test cases. Our previous studies show that switching cost can affect the efficiency of testing. ObjectiveIn this paper, we aim to identify the optimal testing order that can detect interaction triggered faults earlier in the presence switching cost. MethodWe presented a distance based metric to measure the switching cost between test cases. As reducing the switching cost can make the whole test suite run faster and thus achieve full combination coverage earlier, single-objective algorithms were used to minimize the total switching cost. Besides, when determining the next test case to run, there is a trade-off between high combination coverage and low switching cost. Hence, hybrid and multi-objective algorithms were used to achieve a better balance. In order to evaluate different algorithms, we conducted a series of experiments covering 400 different testing scenarios. We also conducted an empirical study with six real world applications. ResultsThe heuristic solver for the travelling salesman problem is the best algorithm to minimize the switching cost. It can detect faults earlier than the order with high rate of combination coverage. But in order to further reduce the effort to detect the first fault, the hybrid and multi-objective algorithms are the best methods. ConclusionPrioritization based on switching cost can speed up the fault detection to some extent, but prioritization based on both combination coverage and testing cost can deliver the optimal testing order.