Abstract:Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

BERT: BEhavioral Regression Testing.

Automated Behavioral Regression Testing

BERT: a tool for behavioral regression testing

A Novel Fuzzy Classification to Enhance Software Regression Testing

Machine Learning and Evolutionary Computing for GUI-based Regression Testing

DiffGen: Automated Regression Unit-Test Generation

DiffGen: Automated Unit Test Generation for Regression Testing

Scenario-Based Functional Regression Testing

Testing Reactive Systems Using Behavioural Programming, a Model Centric Approach

Regression Test Cases Generation Based on Automatic Model Revision

Isomorphic Regression Testing: Executing Uncovered Branches Without Test Augmentation.

An Auditing Test To Detect Behavioral Shift in Language Models

Checking Inside the Black Box: Regression Testing Based on Value Spectra Differences

Dependent-test-aware Regression Testing Techniques

Guided path exploration for regression test generation

Checking Inside the Black Box: Regression Testing by Comparing Value Spectra

Regression Test Generation Approach Based on Tree-Structured Analysis

Erlang Code Evolution Control

Is this a bug or an obsolete test?

ComboRT: A New Approach for Generating Regression Test Cases for Evolving Programs.

Dynamic Analysis can be Improved with Automatic Test Suite Refactoring