Abstract:Although regression testing is important to guarantee the software quality in software evolution, it suffers from the widely known cost problem. To address this problem, existing researchers made dedicated efforts on test prioritization, which optimizes the execution order of tests to detect faults earlier; while practitioners in industry leveraged more computing resources to save the time cost of regression testing. By combining these two orthogonal solutions, in this article, we define the problem of parallel test prioritization, which is to conduct test prioritization in the scenario of parallel test execution to reduce the cost of regression testing. Different from traditional sequential test prioritization, parallel test prioritization aims at generating a set of test sequences, each of which is allocated in an individual computing resource and executed in parallel. In particular, we propose eight parallel test prioritization techniques by adapting the existing four sequential test prioritization techniques, by including and excluding testing time in prioritization. To investigate the performance of the eight parallel test prioritization techniques, we conducted an extensive study on 54 open-source projects and a case study on 16 commercial projects from Baidu, a famous search service provider with 600M monthly active users. According to the two studies, parallel test prioritization does improve the efficiency of regression testing, and cost-aware additional parallel test prioritization technique significantly outperforms the other techniques, indicating that this technique is a good choice for practical parallel testing. Besides, we also investigated the influence of two external factors, the number of computing resources and time allowed for parallel testing, and find that more computing resources indeed improve the performance of parallel test prioritization. In addition, we investigated the influence of two more factors, test granularity and coverage criterion, and find that parallel test prioritization can still accelerate regression testing in parallel scenario. Moreover, we investigated the benefit of parallel test prioritization on the regression testing process of continuous integration, considering both the cumulative acceleration performance and the overhead of prioritization techniques, and the results demonstrate the superiority of parallel test prioritization.

Parallel Technique Analysis for Effective Reducing Test Cost

A Parallel Implementation Strategy of Adaptive Testing

Parallel Test Prioritization.

Accelerating RTL Quality Assessment Method Based on CAD Tools

Effective Design Method to Reduce SCAN Test Application Time of SOC's

Implementation Method of SOC Concurrent Test Based on ATE

The Test Cost Reduction Benefits of Combining a Hierarchical DFT Methodology with EDT Channel Sharing - A Case Study.

Functional Test-Cost Reduction Based on Fault Tree Analysis and Binary Optimization

A Unified Solution to Reduce Test Power and Test Volume for Test-per-scan Schemes

Functional Test-Cost Reduction Based on Optimization Modeling and Congestion Control

Parallel test scheduling optimization method for three-dimensional chip with multi-core and multi-layer

Methods of Sequential Test Optimization in Dynamic Environment

A Novel Hybrid Delay Testing Scheme with Low Test Power, Volume, and Time

On-demand test suite reduction

An Experimental Study of Four Typical Test Suite Reduction Techniques

Parallel testing optimization method of digital microfluidic biochip

A Novel Test Application Scheme for High Transition Fault Coverage and Low Test Cost

Pvt: Unified Reduction Of Test Power, Volume, And Test Time Using Double-Tree Scan Architecture

An Experimental Comparison of Four Test Suite Reduction Techniques.

A Test-Suite Reduction Approach to Improving Fault-Localization Effectiveness

Low-power Technique of Scan-Based Design for Test