Abstract:Developers typically rely on regression testing techniques to ensure that their changes do not break existing functionality. Unfortunately, these techniques suffer from flaky tests, which can both pass and fail when run multiple times on the same version of code and tests. One prominent type of flaky tests is order-dependent (OD) tests, which are tests that pass when run in one order but fail when run in another order. Although OD tests may cause flaky-test failures, OD tests can help developers run their tests faster by allowing them to share resources. We propose to make regression testing techniques dependent-test-aware to reduce flaky-test failures. To understand the necessity of dependent-test-aware regression testing techniques, we conduct the first study on the impact of OD tests on three regression testing techniques: test prioritization, test selection, and test parallelization. In particular, we implement 4 test prioritization, 6 test selection, and 2 test parallelization algorithms, and we evaluate them on 11 Java modules with OD tests. When we run the orders produced by the traditional, dependent-test-unaware regression testing algorithms, 82% of human-written test suites and 100% of automatically-generated test suites with OD tests have at least one flaky-test failure. We develop a general approach for enhancing regression testing algorithms to make them dependent-test-aware, and apply our approach to 12 algorithms. Compared to traditional, unenhanced regression testing algorithms, the enhanced algorithms use provided test dependencies to produce orders with different permutations or extra tests. Our evaluation shows that, in comparison to the orders produced by unenhanced algorithms, the orders produced by enhanced algorithms (1) have overall 80% fewer flaky-test failures due to OD tests, and (2) may add extra tests but run only 1% slower on average. Our results suggest that enhancing regression testing algorithms to be dependent-test-aware can substantially reduce flaky-test failures with only a minor slowdown to run the tests.

Assuring the Model Evolution of Protocol Software Specifications by Regression Testing Process Improvement

Learning Probabilistic Models for Model Checking: an Evolutionary Approach and an Empirical Study

Identify and Update Test Cases when Production Code Changes: A Transformer-Based Approach

Regression Test Cases Generation Based on Automatic Model Revision

Petri Net Based Test Case Generation for Evolved Specification

Should We Learn Probabilistic Models for Model Checking? A New Approach and An Empirical Study.

Automated Behavioral Regression Testing

High-confidence Software Evolution

Applying Interface-Contract Mutation in Regression Testing of Component-Based Software

Machine Learning and Evolutionary Computing for GUI-based Regression Testing

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

ComboRT: A New Approach for Generating Regression Test Cases for Evolving Programs.

Automatic Regression Test Selection Based on Activity Diagrams

Regression mutation testing.

Improving Effectiveness of Automated Software Testing in the Absence of Specifications

Uncovering Unknown System Behaviors in Uncertain Networks with Model and Search-based Testing

Research Advances in Test Suite Augmentation for Regression Testing

Scenario-Based Functional Regression Testing

Validation and Simulation of Software System Evolution Rules Using Software Networks

Mutation Testing in Evolving Systems: Studying the relevance of mutants to code evolution

Dependent-test-aware Regression Testing Techniques