Abstract:Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a set of metamorphic relations among multiple software inputs and their expected outputs, metamorphic testing not only provides a test case generation strategy by constructing new (or follow-up) test cases from some original (or source) test cases, but also a test result verification mechanism through checking the relationship between the outputs of source and follow-up test cases. Many efforts have been made to further improve the cost-effectiveness of metamorphic testing from different perspectives. Some studies attempted to identify “good” metamorphic relations, while other studies were focused on applying effective test case generation strategies especially for source test cases. In this article, we propose improving the cost-effectiveness of metamorphic testing by leveraging the feedback information obtained in the test execution process. Consequently, we develop a new approach, namely feedback-directed metamorphic testing, which makes use of test execution information to dynamically adjust the selection of metamorphic relations and selection of source test cases. We conduct an empirical study to evaluate the proposed approach based on four laboratory programs, one GNU program, and one industry program. The empirical results show that feedback-directed metamorphic testing can use fewer test cases and take less time than the traditional metamorphic testing for detecting the same number of faults. It is clearly demonstrated that the use of feedback information about test execution does help enhance the cost-effectiveness of metamorphic testing. Our work provides a new perspective to improve the efficacy and applicability of metamorphic testing as well as many other software testing techniques.

Integration of Metamorphic Testing with Program Repair Methods Based on Adaptive Search Strategies and Program Equivalence.

METARO(3) : Metamorphic Relation Group for Automatic Program Repair

The Future Can’t Help Fix the Past: Assessing Program Repair in the Wild

An Iterative Metamorphic Testing Technique for Web Services and Case Studies.

Identifying Metamorphic Relations: A Data Mutation Directed Approach

Exploring and Lifting the Robustness of LLM-powered Automated Program Repair with Metamorphic Testing

Application of Metamorphic Testing Monitored by Test Adequacy in a Monte Carlo Simulation Program.

An Interleaving Guided Metamorphic Testing Approach for Concurrent Programs

Feedback-Directed Metamorphic Testing

Metamorphic relation prioritization for effective regression testing

Testing of Component-Based Software: A Metamorphic Testing Methodology

Improve the Quality of ARC Systems Based on the Metamorphic Testing

Test Adequacy for Metamorphic Testing: Criteria, Measurement, and Implication

An Application of Metamorphic Testing for Testing Scientific Software.

Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis

Bottom-up Integration Testing with the Technique of Metamorphic Testing

Enhanced Evolutionary Automated Program Repair by Finer-Granularity Ingredients and Better Search Algorithms

MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases

Tabular-expression-based Method for Constructing Metamorphic Relations.

MR-Scout : Automated Synthesis of Metamorphic Relations from Existing Test Cases

An Empirical Analysis of the Influence of Fault Space on Search-Based Automated Program Repair.