A Unified Test Case Prioritization Approach.

Dan Hao,Lingming Zhang,Lu Zhang,Gregg Rothermel,Hong Mei

DOI: https://doi.org/10.1145/2685614

2014-01-01

Abstract:Test case prioritization techniques attempt to reorder test cases in a manner that increases the rate at which faults are detected during regression testing. Coverage-based test case prioritization...

What problem does this paper attempt to address?

Test Case Prioritization Based on Varying Testing Requirement Priorities and Test Case Costs

Xiaofang Zhang,Changhai Nie,Baowen Xu,Bo Qu

DOI: https://doi.org/10.1109/qsic.2007.4385476

2007-01-01

Abstract:Test case prioritization is an effective and practical technique in regression testing. It schedules test cases in order of precedence that increases their ability to meet some performance goals, such as code coverage, rate of fault detection. In previous work, the test case prioritization techniques and metrics usually assumed that testing requirement priorities and test case costs are uniform. In this paper, basing on varying testing requirement priorities and test case costs, we present a new, general test case prioritization technique and an associated metric. The case study illustrates that the rate of "units-oftesting-requirement-priority-satisfied- per-unit-test-case-cost" can be increased, and then the testing quality and customer satisfaction can be improved.
A Static Approach to Prioritizing JUnit Test Cases

Hong Mei,Dan Hao,Lingming Zhang,Lu Zhang,Ji Zhou,Gregg Rothermel

DOI: https://doi.org/10.1109/TSE.2011.106

IF: 7.4

2012-01-01

IEEE Transactions on Software Engineering

Abstract:Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most of these techniques require data on dynamic execution in the form of code coverage information for test cases. However, the collection of dynamic code coverage information on test cases has several associated drawbacks including cost increases and reduction in prioritization precision. In this paper, we propose an approach to prioritizing test cases in the absence of coverage information that operates on Java programs tested under the JUnit framework-an increasingly popular class of systems. Our approach, JUnit test case Prioritization Techniques operating in the Absence of coverage information (JUPTA), analyzes the static call graphs of JUnit test cases and the program under test to estimate the ability of each test case to achieve code coverage, and then schedules the order of these test cases based on those estimates. To evaluate the effectiveness of JUPTA, we conducted an empirical study on 19 versions of four Java programs ranging from 2K-80K lines of code, and compared several variants of JUPTA with three control techniques, and several other existing dynamic coverage-based test case prioritization techniques, assessing the abilities of the techniques to increase the rate of fault detection of test suites. Our results show that the test suites constructed by JUPTA are more effective than those in random and untreated test orders in terms of fault-detection effectiveness. Although the test suites constructed by dynamic coverage-based techniques retain fault-detection effectiveness advantages, the fault-detection effectiveness of the test suites constructed by JUPTA is close to that of the test suites constructed by those techniques, and the fault-detection effectiveness of the test suites constructed by some of - UPTA's variants is better than that of the test suites constructed by several of those techniques.
Test-case Prioritization: Achievements and Challenges.

Dan Hao,Lu Zhang,Hong Mei

DOI: https://doi.org/10.1007/s11704-016-6112-3

IF: 2.6688

2016-01-01

Frontiers of Computer Science

Abstract:Test-case prioritization, proposed at the end of last century, aims to schedule the execution order of test cases so as to improve test effectiveness. In the past years, test-case prioritization has gained much attention, and has significant achievements in five aspects: prioritization algorithms, coverage criteria, measurement, practical concerns involved, and application scenarios. In this article, we will first review the achievements of test-case prioritization from these five aspects and then give our perspectives on its challenges.
Prioritizing JUnit test cases in absence of coverage information

Lingming Zhang,Ji Zhou,Dan Hao,Lu Zhang,Hong Mei

DOI: https://doi.org/10.1109/ICSM.2009.5306350

2009-01-01

Abstract:Better orderings of test cases can detect faults in less time with fewer resources, and thus make the debugging process earlier and accelerate software delivery. As a result, test case prioritization has become a hot topic in the research of regression testing. With the popularity of using the JUnit testing framework for developing Java software, researchers also paid attention to techniques for prioritizing JUnit test cases in regression testing of Java software. Typically, most of them are based on coverage information of test cases. However, coverage information may need extra costs to acquire. In this paper, we propose an approach (named Jupta) for prioritizing JUnit test cases in absence of coverage information. Jupta statically analyzes call graphs of JUnit test cases and the software under test to estimate the test ability (TA) of each test case. Furthermore, Jupta provides two prioritization techniques: the total TA based technique (denoted as JuptaT) and the additional TA based technique (denoted as JuptaA). To evaluate Jupta, we performed an experimental study on two open source Java programs, containing 11 versions in total. The experimental results indicate that Jupta is more effective and stable than the untreated orderings and Jupta is approximately as effective and stable as prioritization techniques using coverage information at the method level.
Adaptive Test-Case Prioritization Guided by Output Inspection

Dan Hao,Xu Zhao,Lu Zhang

DOI: https://doi.org/10.1109/COMPSAC.2013.31

2013-01-01

Abstract:Test-case prioritization is to schedule the execution order of test cases so as to maximize some objective (e.g., revealing faults early). The existing test-case prioritization approaches separate the process of test-case prioritization and the process of test-case execution by presenting the execution order of all test cases before programmers start running test cases. As the execution information of the modified program is not available for the existing test-case prioritization approaches, these approaches mainly rely on only the execution information of the previous program before modification. To address this problem, we present an adaptive test-case prioritization approach, which determines the execution order of test cases simultaneously during the execution of test cases. In particular, the adaptive approach selects test cases based on their fault-detection capability, which is calculated based on the output of selected test cases. As soon as a test case is selected and runs, the fault-detection capability of each unselected test case is modified according to the output of the latest selected test case. To evaluate the effectiveness of the proposed adaptive approach, we conducted an experimental study on eight C programs and four Java programs. The experimental results show that the adaptive approach is usually significantly better than the total test-case prioritization approach and competitive to the additional test-case prioritization approach. Moreover, the adaptive approach is better than the additional approach on some subjects (e.g, replace and schedule).
Test Case Prioritization Incorporating Ordered Sequence of Program Elements

Kun Wu,Chunrong Fang,Zhenyu Chen,Zhihong Zhao

DOI: https://doi.org/10.1109/iwast.2012.6228980

2012-01-01

Abstract:Test suites often grow very large over many releases, such that it is impractical to re-execute all test cases within limited resources. Test case prioritization, which rearranges test cases, is a key technique to improve regression testing. Code coverage information has been widely used in test case prioritization. However, other important information, such as the ordered sequence of program elements measured by execution frequencies, was ignored by previous studies. It raises a risk to lose detections of difficult-to-find bugs. Therefore, this paper improves the similarity-based test case prioritization using the ordered sequence of program elements measured by execution counts. The empirical results show that our new technique can increase the rate of fault detection more significantly than the coverage-based ART technique. Moreover, our technique can detect bugs in loops more quickly and be more cost-benefits than the traditional ones.
Test Case Prioritization for Black Box Testing

Bo Qu,Changhai Nie,Baowen Xu,Xiaofang Zhang

DOI: https://doi.org/10.1109/COMPSAC.2007.209

2007-01-01

Abstract:Test case prioritization is an effective and practical technique that helps to increase the rate of regression fault detection when software evolves. Numerous techniques have been reported in the literature on prioritizing test cases for regression testing. However, existing prioritization techniques implicitly assume that source or binary code is available when regression testing is performed, and therefore cannot be implemented when there is no program source or binary code to be analyzed. In this paper, we presented a new technique for black box regression testing, and we performed an experiment to measure our technique. Our results show that the new technique is helpful to improve the effectiveness of fault detection when performing regression test in black box environment.
A Dynamic Prioritization Method for Supplemental Test Cases

Changhai Nie

2009-01-01

Abstract:The test case prioritization is an effective technique of regression testing. However, the existing techniques mainly focus on the reused test suite, and ignored the supplemental test cases. To solve this problem, a new dynamic prioritization method was presented for supplemental test cases by extending the usage of previous prioritization techniques. Also, its implementation in the parallel scenario was discussed. A case study was used to indicate the conclusion that the new method could improve the effectiveness of the supplemental test cases.
Time-Aware Test-Case Prioritization Using Integer Linear Programming

Lu Zhang,Shan-Shan Hou,Chao Guo,Tao Xie,Hong Mei

DOI: https://doi.org/10.1145/1572272.1572297

2009-01-01

Abstract:Techniques for test-case prioritization re-order test cases to increase their rate of fault detection. When there is a fixed time budget that does not allow the execution of all the test cases, time-aware techniques for test-case prioritization may achieve a better rate of fault detection than traditional techniques for test-case prioritization. In this paper, we propose a novel approach to time-aware test-case prioritization using integer linear programming. To evaluate our approach, we performed experiments on two subject programs involving four techniques for our approach, two techniques for an approach to time-aware test-case prioritization based on genetic algorithms, and four traditional techniques for test-case prioritization. The empirical results indicate that two of our techniques outperform all the other techniques for the two subjects under the scenarios of both general and version-specific prioritization. The empirical results also indicate that some traditional techniques with lower analysis time cost for test-case prioritization may still perform competitively when the time budget is not quite tight.
Multi-objective Test Case Prioritization for GUI Applications.

Wei Sun,Zebao Gao,Weiran Yang,Chunrong Fang,Zhenyu Chen

DOI: https://doi.org/10.1145/2480362.2480566

2013-01-01

Abstract:Test case prioritization techniques are proposed to schedule execution of test cases in order to improve testing effectiveness. Various coverage criteria are used as surrogates for test case prioritization. They are expected to improve testing effectiveness by satisfying the surrogates as early as possible. In test case prioritization, statement coverage is widely used for conventional applications and event coverage is applied for GUI applications. GUI applications are different from conventional applications for interactions between front-end events and back-end codes. Such complex interactions lead to the single-objective strategy insufficient for testing of GUI applications in many cases. In this paper, we propose a multi-objective test case prioritization strategy to combine event coverage and statement coverage for GUI applications. The preliminary experimental results show that our multi-objective strategy can outperform the single-objective strategies.
To Be Optimal or Not in Test-Case Prioritization.

Dan Hao,Lu Zhang,Lei Zang,Yanbo Wang,Xingxia Wu,Tao Xie

DOI: https://doi.org/10.1109/tse.2015.2496939

IF: 7.4

2015-01-01

IEEE Transactions on Software Engineering

Abstract:Software testing aims to assure the quality of software under test. To improve the efficiency of software testing, especially regression testing, test-case prioritization is proposed to schedule the execution order of test cases in software testing. Among various test-case prioritization techniques, the simple additional coverage-based technique, which is a greedy strategy, achieves surprisingly competitive empirical results. To investigate how much difference there is between the order produced by the additional technique and the optimal order in terms of coverage, we conduct a study on various empirical properties of optimal coverage-based test-case prioritization. To enable us to achieve the optimal order in acceptable time for our object programs, we formulate optimal coverage-based test-case prioritization as an integer linear programming (ILP) problem. Then we conduct an empirical study for comparing the optimal technique with the simple additional coverage-based technique. From this empirical study, the optimal technique can only slightly outperform the additional coverage-based technique with no statistically significant difference in terms of coverage, and the latter significantly outperforms the former in terms of either fault detection or execution time. As the optimal technique schedules the execution order of test cases based on their structural coverage rather than detected faults, we further implement the ideal optimal test-case prioritization technique, which schedules the execution order of test cases based on their detected faults. Taking this ideal technique as the upper bound of test-case prioritization, we conduct another empirical study for comparing the optimal technique and the simple additional technique with this ideal technique. From this empirical study, both the optimal technique and the additional technique significantly outperform the ideal technique in terms of coverage, but the latter significantly outperforms the former two techniques in terms of fault detection. Our findings indicate that researchers may need take cautions in pursuing the optimal techniques in test-case prioritization with intermediate goals.
Test Case Prioritization for Multiple Processing Queues

Bo Qu,Changhai Nie,Baowen Xu

DOI: https://doi.org/10.1109/isise.2008.106

2008-01-01

Abstract:Test case prioritization is an effective technique that helps to increase the rate of fault detection or code coverage in regression testing. However, all existing methods can only prioritize test cases to a single queue. Once there are two or more machines that participate in testing, all exiting techniques are not applicable any more. To extend the prioritization methods to parallel scenario, this paper defines the prioritization problem in such scenario and applies the task scheduling method to prioritization algorithms to help partitioning a test suite into multiple prioritized subsets. Besides, this paper also discusses the limitation of previous metrics and proposes a new measure of effectiveness of prioritization methods in a parallel scenario. Finally, a case study is performed to illustrate the algorithms and metrics presented in this article.
Test Scenario Prioritization Using UML Use Case and Activity Diagram

Prachet Bhuyan,Abhishek Ray,Manali Das

DOI: https://doi.org/10.1007/978-981-10-3874-7_47

2017-01-01

Abstract:Software testing mainly aims at providing software quality assurance by verifying the behavior of a software using a finite set of test cases. The continuous evolution of software makes it impossible to perform exhaustive testing. The need for regression testing is to uncover new software bugs in existing system after some changes have been made to ensure that the existing functionalities are working fine. Re-executing the whole test suite is time-consuming as well as expensive. Hence, this issue can be handled by test case prioritization technique. Prioritization helps to organize the test suites in an effective manner where high-priority test cases are executed earlier than the low priority test cases based on some criteria. In this paper, a new prioritization approach is proposed using UML use case diagram and UML activity diagram. We have applied our technique to a particular of a case study which indicates the effectiveness of our proposed approach in prioritizing test scenarios.
Defect-Oriented Test Case Prioritization

Jinfeng Li,Yan Zhang,Jilong Bian

DOI: https://doi.org/10.1007/978-981-32-9686-2_72

2019-01-01

Abstract:In order to find more software defects during software testing within a short period of time, test case prioritization by ordering test cases is studied. This paper introduces four influence factors: defect detection number, defect detection rate, defect rank number and defect rank detection rate. In order to facilitate the tester to balance the four influence factors according to testing requirement, four weight factors are introduced. Meanwhile, we developed an on-line adjustment strategy to adjust the priorities of test cases. Experimental results show that compared with the traditional test case prioritization, the proposed method can find more defects within a short period of time and improve the testing efficiency.
Bridging the gap between the total and additional test-case prioritization strategies

Lingming Zhang,Dan Hao,Lu Zhang,Gregg Rothermel,Hong Mei

DOI: https://doi.org/10.1109/ICSE.2013.6606565

2013-01-01

Abstract:In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total numbers of elements covered per test, and numbers of additional (not-yet-covered) elements covered per test, are two widely-adopted generic strategies used for such prioritization. This paper proposes a basic model and an extended model that unify the total strategy and the additional strategy. Our models yield a spectrum of generic strategies ranging between the total and additional strategies, depending on a parameter referred to as the p value. We also propose four heuristics to obtain differentiated p values for different methods under test. We performed an empirical study on 19 versions of four Java programs to explore our results. Our results demonstrate that wide ranges of strategies in our basic and extended models with uniform p values can significantly outperform both the total and additional strategies. In addition, our results also demonstrate that using differentiated p values for both the basic and extended models with method coverage can even outperform the additional strategy using statement coverage.
Test Case Prioritization Using Partial Attention

Quanjun Zhang,Chunrong Fang,Weisong Sun,Shengcheng Yu,Yutao Xu

DOI: https://doi.org/10.48550/arXiv.2206.11026

2022-06-22

Abstract:Test case prioritization (TCP) aims to reorder the regression test suite with a goal of increasing the fault detection rate. Various TCP techniques have been proposed based on different prioritization strategies. Among them, the greedy-based techniques are the most widely-used TCP techniques. However, existing greedy-based techniques usually reorder all candidate test cases in prioritization iterations, resulting in both efficiency and effectiveness problems. In this paper, we propose a generic partial attention mechanism, which adopts the previous priority values (i.e., the number of additionally-covered code units) to avoid considering all candidate test cases. Incorporating the mechanism with the additional-greedy strategy, we implement a novel coverage-based TCP technique based on partition ordering (OCP). OCP first groups the candidate test cases into different partitions and updates the partitions on the descending order. We conduct a comprehensive experiment on 19 versions of Java programs and 30 versions of C programs to compare the effectiveness and efficiency of OCP with six state-of-the-art TCP techniques: total-greedy, additional-greedy, lexicographical-greedy, unify-greedy, art-based, and search-based. The experimental results show that OCP achieves a better fault detection rate than the state-of-the-arts. Moreover, the time costs of OCP are found to achieve 85%-99% improvement than most state-of-the-arts.

Software Engineering
Test Case Prioritization for Compilers: A Text-Vector Based Approach

Junjie Chen,Dan Hao,Y. Bai,Lu Zhang,Bing Xie,Yingfei Xiong,Hongyu Zhang

DOI: https://doi.org/10.1109/ICST.2016.19

2016-04-11

Abstract:Test case prioritization aims to schedule the execution order of test cases so as to detect bugs as early as possible. For compiler testing, the demand for both effectiveness and efficiency imposes challenge to test case prioritization. In the literature, most existing approaches prioritize test cases by using some coverage information (e.g., statement coverage or branch coverage), which is collected with considerable extra effort. Although input-based test case prioritization relies only on test inputs, it can hardly be applied when test inputs are programs. In this paper we propose a novel text-vector based test case prioritization approach, which prioritizes test cases for C compilers without coverage information. Our approach first transforms each test case into a text-vector by extracting its tokens which reflect fault-relevant characteristics and then prioritizes test cases based on these text-vectors. In particular, in our approach we present three prioritization strategies: greedy strategy, adaptive random strategy, and search strategy. To investigate the efficiency and effectiveness of our approach, we conduct an experiment on two C compilers (i.e., GCC and LLVM), and find that our approach is much more efficient than the existing approaches and is effective in prioritizing test cases.

Computer Science
COST-COGNIZANT COMBINATORIAL TEST CASE PRIORITIZATION

Ziyuan Wang,Lin Chen,Baowen Xu,Yan Huang

DOI: https://doi.org/10.1142/s0218194011005499

IF: 1.007

2011-01-01

International Journal of Software Engineering and Knowledge Engineering

Abstract:Combinatorial testing has been widely used in practice. People usually assume all test cases in combinatorial test suite will run completely. However, in many scenarios where combinatorial testing is needed, for example the regression testing, the entire combinatorial test suite is not run completely as a result of test resource constraints. To improve the efficiency of testing, combinatorial test case prioritization technique is required. For the scenario of regression testing, this paper proposes a new cost-cognizant combinatorial test case prioritization technique, which takes both combination weights and test costs into account. Here we propose a series of metrics with physical meaning, which assess the combinatorial coverage efficiency of test suite, to guide the prioritization of combinatorial test cases. And two heuristic test case prioritization algorithms, which are based on total and additional techniques respectively, are utilized in our technique. Simulation experimental results illustrate some properties and advantages of proposed technique.
Survey of Test Case Prioritization Techniques for Regression Testing

CHEN Xiang,CHEN Ji-Hong,JU Xiao-Lin,GU Qing

DOI: https://doi.org/10.3724/sp.j.1001.2013.04420

2017-01-01

Indian Journal of Science and Technology

Abstract:Test case prioritization (TCP) issue is a hot research topic in regression testing research. This method tries to optimize the execution schedule based on a specific prioritization criterion. The purpose of the TCP techniques is to maximize a specific prioritization objective, such as the early fault detection rate of the original test suite. This technique is especially applied to some testing scenarios, for example testing resource is limited for executing all the test cases. This paper first describes the issue of TCP and classifies the existing TCP techniques into three categories: source code, requirement, and model. The paper secondly formulates a specific TCP issue (i.e., resource-aware TCP issue) and summarizes its research work. The paper finally summarizes commonly-used evaluation metrics and subjects in experimental studies, and empirical result affection of different fault injection types. The paper fourthly summarizes the application of TCP in some specific testing domains, such as combinatorial testing, event-driven applications testing, fault localization, and Web services testing and discusses some future work of the TCP issue.
PRIORITIZATION OF COMBINATORIAL TEST CASES BY INCREMENTAL INTERACTION COVERAGE

Rubing Huang,Xiaodong Xie,Dave Towey,Tsong Yueh Chen,Yansheng Lu,Jinfu Chen

DOI: https://doi.org/10.1142/s0218194013500459

IF: 1.007

2013-01-01

International Journal of Software Engineering and Knowledge Engineering

Abstract:Combinatorial interaction testing is a well-recognized testing method, and has been widely applied in practice, often with the assumption that all test cases in a combinatorial test suite have the same fault detection capability. However, when testing resources are limited, an alternative assumption may be that some test cases are more likely to reveal failure, thus making the order of executing the test cases critical. To improve testing cost-effectiveness, prioritization of combinatorial test cases is employed. The most popular approach is based on interaction coverage, which prioritizes combinatorial test cases by repeatedly choosing an unexecuted test case that covers the largest number of uncovered parameter value combinations of a given strength (level of interaction among parameters). However, this approach suffers from some drawbacks. Based on previous observations that the majority of faults in practical systems can usually be triggered with parameter interactions of small strengths, we propose a new strategy of prioritizing combinatorial test cases by incrementally adjusting the strength values. Experimental results show that our method performs better than the random prioritization technique and the technique of prioritizing combinatorial test suites according to test case generation order, and has better performance than the interaction-coverage-based test prioritization technique in most cases.

A Unified Test Case Prioritization Approach.

Test Case Prioritization Based on Varying Testing Requirement Priorities and Test Case Costs

A Static Approach to Prioritizing JUnit Test Cases

Test-case Prioritization: Achievements and Challenges.

Prioritizing JUnit test cases in absence of coverage information

Adaptive Test-Case Prioritization Guided by Output Inspection

Test Case Prioritization Incorporating Ordered Sequence of Program Elements

Test Case Prioritization for Black Box Testing

A Dynamic Prioritization Method for Supplemental Test Cases

Time-Aware Test-Case Prioritization Using Integer Linear Programming

Multi-objective Test Case Prioritization for GUI Applications.

To Be Optimal or Not in Test-Case Prioritization.

Test Case Prioritization for Multiple Processing Queues

Test Scenario Prioritization Using UML Use Case and Activity Diagram

Defect-Oriented Test Case Prioritization

Bridging the gap between the total and additional test-case prioritization strategies

Test Case Prioritization Using Partial Attention

Test Case Prioritization for Compilers: A Text-Vector Based Approach

COST-COGNIZANT COMBINATORIAL TEST CASE PRIORITIZATION

Survey of Test Case Prioritization Techniques for Regression Testing

PRIORITIZATION OF COMBINATORIAL TEST CASES BY INCREMENTAL INTERACTION COVERAGE