Abstract:Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

Exploiting Synergy Between Testing and Inferred Partial Specifications

Mutually Enhancing Test Generation and Specification Inference

Learning Likely Invariants to Explain Why a Program Fails

Tool-assisted Unit-Test Generation and Selection Based on Operational Abstractions

Combining Model Checking and Testing with an Application to Reliability Prediction and Distribution

Improving Effectiveness of Automated Software Testing in the Absence of Specifications

Directed test suite augmentation via exploiting program dependency.

Tool-assisted Unit Test Selection Based on Operational Violations

Automatically Identifying Special and Common Unit Tests Based on Inferred Statistical Algebraic Abstractions

Specification Inference for Evolving Systems

Augmenting automatically generated unit-test suites with regression oracle checking

Automatic identification of common and special object-oriented unit tests.

Automatic Software Diversity in the Light of Test Suites

Isomorphic Regression Testing: Executing Uncovered Branches Without Test Augmentation.

PreInfer: Automatic Inference of Preconditions via Symbolic Analysis

SpecTest: Specification-Based Compiler Testing

Guided Test Generation for Database Applications Via Synthesized Database Interactions.

What Good Are Strong Specifications?

Testing by Dualization

Dependent-test-aware Regression Testing Techniques

Test Selection for Result Inspection Via Mining Predicate Rules