Abstract:Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

Test Data Reduction Based on Dominance Relations of Target Statements

Regression Test Case Reduction Based on Mutation Analysis and Coverage Criterion

Generalized Decision Reduction and Upper Approximation Reduction Based on Dominance Relation

A study of relative redundancy in test-suite reduction while retaining or improving fault-localization effectiveness.

Research on Test Suite Reduction Using Attribute Relevance Analysis.

Automated Evolutionary Test Data Generation with Domain Reduction for Aspect-Oriented Programs

Upper Approximation Reduction in Inconsistent Target Information System Based on Dominance Relations

On-demand test suite reduction

Matrix Computation for Assignment Reduction in Information Systems Based on Dominance Relations

Improving Generation of Object-Oriented Test Suites by Avoiding Redundant Tests

Test Suite Reduction via Evolutionary Clustering

Dominance Analysis and Its Application

Discernibility Matrix with Respect to Reduction Based on Dominance Relation in Fuzzy Decision Information System

A Method of Test Case Set Generation in the Commutativity Test of Reduce Functions

An Experimental Study of Four Typical Test Suite Reduction Techniques

A Test-Suite Rceduction Based on Multi-Objective Decision Making

Dominance-Based Neighborhood Rough Sets and Its Attribute Reduction.

Distribution Reduction in Inconsistent Information Systems Based on Dominance Relations

A Strengthened Dominance Relation Considering Convergence and Diversity for Evolutionary Many-Objective Optimization

An Evolutionary Generation Method of Test Data for Multiple Paths Based on Coverage Balance

Dependent-test-aware Regression Testing Techniques