Adaptive software testing with fixed-memory feedback

CaiKai-Yuan,GuBo,HuHai,LiYong-Chao

IF: 3.5

2007-01-01

Journal of Systems and Software

Abstract:Adaptive software testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted on-line by using the testing data collected during s...

What problem does this paper attempt to address?

Adaptive Software Testing with Fixed-Memory Feedback.

Kai-Yuan Cai,Bo Gu,Hai Hu,Yong-Chao Li

DOI: https://doi.org/10.1016/j.jss.2006.11.008

2004-01-01

Abstract:Adaptive software testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted on-line by using the testing data collected during software testing as our understanding of the software under test is improved. In this paper we propose a new strategy of adaptive software testing in the context of software cybernetics. This new strategy employs fixed-memory feedback for on-line parameter estimations and is intended to circumvent the drawbacks of the assumption that all remaining defects are equally detectable at constant rate and to reduce the underlying computational complexity of on-line parameter estimations. A comprehensive case study with the Space program demonstrates that the new adaptive testing strategy can really work in practice and may noticeably outperform the purely-random testing strategy and the random-partition testing strategy (or collectively, the random testing strategies) in terms of the number of tests used to detect and remove a given number of defects in a single process of software testing and the corresponding standard deviation. In addition, the case study shows that the input domain of the software under test should be partitioned non-evenly for the adaptive testing strategy.
A Case Study For Determining Tile Optimal, Size Of Fixed-Memory Feedback In Adaptive Software Testing

Y Shi,Bb Yin,Ch Jiang,H Hu,Cg Bai,Ky Cai

2005-01-01

Abstract:Adaptive software testing refers to the testing strategy that can adjust itself on-line based on the feedback history information. The amount of feedback information of adaptive testing strategy has an essential influence on the performance of testing. In this paper, how the size of fixed-memory feedback affects the performance of adaptive testing is studied and an estimation of the optimal size is presented. A series of controlled software testing experiments are implemented on three subject programs, a C program named "YaBasic" which acts as a Basic language interpreter, the Space program and a measurement program named "SESD (Software Environment for Software Science Data Collection)". Defects were intentionally injected into the subject program and adaptive testing strategies with different amount of feedback information are applied to the subject program to find the injected defects. Testing results show that adaptive strategies with different sizes of fixed-memory feedback perform differently in terms of the number of test cases used to detect and remove a given number of defects in a single process of testing and the corresponding standard deviation. There exists an optimal size of fixed-memory feedback in adaptive strategy. Both over-large and over-small sizes will depress the performance of adaptive testing. Moreover, our study also reveals that the optimal window size is close to the length of the second stage of the testing process.
An experimental study of adaptive testing for software reliability assessment

CaiKai-Yuan,JiangChang-Hai,HuHai,BaiCheng-Gang

IF: 3.5

2008-01-01

Journal of Systems and Software

Abstract:Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previ...
An Improved Approach to Adaptive Testing

Hai Hu,Chang-Hai Jiang,Kai-Yuan Cai

DOI: https://doi.org/10.1142/s0218194009004349

IF: 1.007

2009-01-01

International Journal of Software Engineering and Knowledge Engineering

Abstract:Adaptive testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted on-line by using the testing data collected during software testing as our understanding of the software under test is improved. Previous studies on adaptive testing involved a simplified Controlled Markov Chain (CMC) model for software testing which employs several unrealistic assumptions. In this paper we propose a new adaptive software testing approach in the context of an improved and namely, general CMC model which aims to eliminate such threats to validity. A set of more realistic basic assumptions on the software testing process is proposed and several unrealistic assumptions are replaced by less unrealistic assumptions. A new adaptive testing strategy based on the general CMC is developed and implemented. Mathematical simulations and experiments on real life software are conducted to demonstrate the effectiveness of the new strategy.
Adaptive Software Testing in the Context of an Improved Controlled Markov Chain Model

Hai Hu,Chang-Hai Jiang,Kai-Yuan Cai

DOI: https://doi.org/10.1109/COMPSAC.2008.186

2008-01-01

Abstract:Adaptive software testing is the counterpart of adaptive control in software testing. It means that software testing strategy should be adjusted online by using the testing data collected during software testing as our understanding of the software under test improves. Previous studies on adaptive testing rely on a simplified Controlled Markov Chain (CMC) model for software testing which employs several unrealistic assumptions. In this paper we propose a new adaptive software testing approach in the context of an improved CMC model which aims to eliminate such threats to validity. A new set of basic assumptions on the software testing process is proposed and several unrealistic assumptions are replaced by more common situations in real life software testing. The methodology of a new adaptive testing strategy is also developed and implemented. Experimental data are collected to demonstrate the effectiveness of the new methodology.
Optimal and Adaptive Testing for Software Reliability Assessment

KY Cai,YC Li,K Liu

DOI: https://doi.org/10.1016/j.infsof.2004.07.006

IF: 3.9

2004-01-01

Information and Software Technology

Abstract:Optimal software testing is concerned with how to test software such that the underlying testing goal is achieved in an optimal manner. Our previous work shows that the optimal testing problem for software reliability growth can be treated as closed-loop or feedback control problem, where the software under test serves as a controlled object and the software testing strategy serves as the corresponding controller. More specifically, the software under test is modeled as controlled Markov chains (CMCs) and the control theory of Markov chains is used to synthesize the required optimal testing strategy. In this paper, we show that software reliability assessment can be treated as a feedback control problem and the CMC approach is also applicable to dealing with the optimal testing problem for software reliability assessment. In this problem, the code of the software under test is frozen and the software testing process is optimized in the sense that the variance of the software reliability estimator is minimized. An adaptive software testing strategy is proposed that uses the testing data collected on-line to estimate the required parameters and selects next test cases. Simulation results show that the proposed adaptive software testing strategy can really work in the sense that the resulting variance of the software reliability estimate is much smaller than that resulting from the random testing strategies. The work presented in this paper is a contribution to the new area of software cybernetics that explores the interplay between software and control.
Adaptive testing of software components

Kai-Yuan Cai,T. Y. Chen,Yong-Chao Li,Wei-Yi Ning,Y. T. Yu

DOI: https://doi.org/10.1145/1066677.1067011

2005-01-01

Abstract:Software components are popular in nowadays software industries. However, how to test software components is a problem since the source code of the software component under test may not be available for the third-party user. In this paper we show that the software component should be tested in an adaptive manner in the sense that the software defect detection rates are estimated on-line by using testing data collected during testing to improve test case selections. In doing so, we use a recursive least squares estimation method to do on-line parameter estimations. This paper further justifies the advantages of the controlled Markov chain (CMC) approach to software testing in particular, and the practicality of the idea of software cybermetics in general.
Optimal software testing and adaptive software testing in the context of software cybernetics

Kai-Yuan Cai

DOI: https://doi.org/10.1016/S0950-5849(02)00108-8

IF: 3.9

2002-01-01

Information and Software Technology

Abstract:Software cybernetics explores the interplay between software theory/engineering and control theory/engineering. Following the idea of software cybernetics, the controlled Markov chains (CMC) approach to software testing treats software testing as a control problem. The software under test serves as a controlled object, and the (optimal) testing strategy determined by the theory of controlled Markov chains serves as a controller. This paper analyzes the behavior of the corresponding optimal test profile determined by the CMC approach to software testing and introduces adaptive software testing. It is shown that in some cases the optimal test profile is Markovian, whereas in some other cases the optimal test profile demonstrates a different scenario. The adaptive software testing adjusts software testing strategy on-line by using testing data collected during software testing in response to changes in our understanding of the software under test. Simulation results show that the adaptive strategy of software testing is feasible and significantly reduces the number of test cases required to detect and remove a certain number of software defects in comparison with the random strategy of software testing.
A self-adaptive test framework for concurrent programs

Manna Wu,Bo Zhou,Wei Shi

DOI: https://doi.org/10.1145/1643823.1643911

2009-01-01

Abstract:In this paper, we describe a new test framework for concurrent programs. Testing on concurrent programs remains difficult due to the non-deterministic nature of concurrent programs. Approaches are proposed to surmount this difficulty. However, few of them are really effective. A new self-adaptive test framework is proposed to alleviate this problem, since it is impossible to solve it completely. This framework which is based on the study of concurrent test points tests each concurrent program like a circuit, while concurrent test points like components of the circuit. The original reason of concurrent bugs stems from the resource sharing among multiple processes or threads. Therefore, a controller is introduced in the framework to increasingly intensify resource competition via test data controlling, aiming to trigger concurrent bugs' emerging sooner once they exist.
Design and Application of a Response-Driven Adaptive Test System

JianPing Zhang,Bo Mu,Shipin Chen

DOI: https://doi.org/10.1109/ITCS.2009.87

2009-01-01

Abstract:Unlike the traditional paper-and-pencil tests, computer adaptive testing (CAT) presents the next item based on his or her past responses, which is more appropriate for a particular individual. However, it is rarely seen in a smaller-scale scenario, such as in classrooms or business, because of the complexity of Item Response Theory (IRT) and the possible usages of services may not be known or may increase due to the development of new services. Our approach to this problem is self-adaptive functional testing, which is a combination of monitoring and active testing at runtime. We present a Response-Driven Adaptive Test System (RDATS) to get userpsilas response data which has been detected during monitoring, existing test cases are adapted automatically to the specific situation and applied afterwards based on userpsilas response at runtime. In this paper, we discuss the structure and functionality of the system with particular emphasis on adaptive test Mechanisms and SQL database design. Finally, we discuss some practical issues related to the application of the system and its future works.
Optimal and Adaptive Testing with Cost Constraints.

Kai-Yuan Cai,Yong-Chao Li,Wei-Yi Ning,W. Eric Wong,Hai Hu

DOI: https://doi.org/10.1145/1138929.1138944

2006-01-01

Abstract:This paper generalizes our previous work on optimal and adaptive testing to consider a more general scenario of software testing resource constraints. The assumption is that software testing must be stopped once the allowed testing resources are used up. The contributions of this paper are as follows. First, we show that software testing with fixed resource constraints can be handled in the framework of the controlled Markov chains (CMC) approach to software testing. Second, an algorithm is adopted to reduce the computational complexity of on-line decision making in optimal and adaptive testing. Finally, the simulation results presented in this paper further confirm the effectiveness of the idea of adaptive testing in particular, and that of software cybernetics (which explores the interplay between software and control) in general.
On The Online Parameter Estimation Problem In Adaptive Software Testing

Kai-Yuan Cai,Tsong Yueh Chen,Yong-Chao Li,Yuen-Tak Yu,Lei Zhao

DOI: https://doi.org/10.1142/S0218194008003696

IF: 1.007

2008-01-01

International Journal of Software Engineering and Knowledge Engineering

Abstract:Software cybernetics is an emerging area that explores the interplay between software and control. The controlled Markov chain (CMC) approach to software testing supports the idea of software cybernetics by treating software testing as a control problem, where the software under test serves as a controlled object modeled by a controlled Markov chain and the software testing strategy serves as the corresponding controller. The software under test and the corresponding software testing strategy form a closed-loop feedback control system. The theory of controlled Markov chains is used to design and optimize the testing strategy in accordance with the testing/reliability goal given explicitly and a priori. Adaptive software testing adjusts and improves software testing strategy online by using the testing data collected in the course of software testing. In doing so, the online parameter estimations play a key role. In this paper, we study the effects of genetic algorithm and the gradient method for doing online parameter estimation in adaptive software testing. We find that genetic algorithm is effective and does not require prior knowledge of the software parameters of concern. Although genetic algorithm is computationally intensive, it leads the adaptive software testing strategy to an optimal software testing strategy that is determined by optimizing a given testing goal, such as minimizing the total cost incurred for removing a given number of defects. On the other hand, the gradient method is computationally favorable, but requires appropriate initial values of the software parameters of concern. It may lead, or fail to lead, the adaptive software testing strategy to an optimal software testing strategy, depending on whether the given initial parameter values are appropriate or not. In general, the genetic algorithm should be used instead of the gradient method in adaptive software testing. Simulation results show that adaptive software testing does work and outperforms random testing.
Adaptive Partition Testing.

Chang-Ai Sun,Hepeng Dai,Huai Liu,Tsong Yueh Chen,Kai-Yuan Cai

DOI: https://doi.org/10.1109/tc.2018.2866040

2019-01-01

Abstract:Random testing and partition testing are two major families of software testing techniques. They have been compared both theoretically and empirically in numerous studies for decades, and it has been widely acknowledged that they have their own advantages and disadvantages and that their innate characteristics are fairly complementary to each other. Some work has been conducted to develop advanced testing techniques through the integration of random testing and partition testing, attempting to preserve the advantages of both while minimizing their disadvantages. In this paper, we propose a new testing approach, adaptive partition testing, where test cases are randomly selected from some partition whose probability of being selected is adaptively adjusted along the testing process. We particularly develop two algorithms, Markov-chain based adaptive partition testing and reward-punishment based adaptive partition testing, to implement the proposed approach. The former algorithm makes use of Markov matrix to dynamically adjust the probability of a partition to be selected for conducting tests; while the latter is based on a reward and punishment mechanism. We conduct empirical studies to evaluate the performance of the proposed algorithms using ten faulty versions of three large-scale open source programs. Our experimental results show that, compared with two baseline techniques, namely random partition testing (RPT) and dynamic random testing (DRT), our algorithms deliver higher fault-detection effectiveness with lower test case selection overhead. It is demonstrated that the proposed adaptive partition testing is an effective testing approach, taking advantages of both random testing and partition testing.
On the Asymptotic Behavior of Adaptive Testing Strategy for Software Reliability Assessment

Junpeng Lv,Bei-Bei Yin,Kai-Yuan Cai

DOI: https://doi.org/10.1109/tse.2014.2310194

IF: 7.4

2014-01-01

IEEE Transactions on Software Engineering

Abstract:In software reliability assessment, one problem of interest is how to minimize the variance of reliability estimator, which is often considered as an optimization goal. The basic idea is that an estimator with lower variance makes the estimates more predictable and accurate. Adaptive Testing (AT) is an online testing strategy, which can be adopted to minimize the variance of software reliability estimator. In order to reduce the computational overhead of decision-making, the implemented AT strategy in practice deviates from its theoretical design that guarantees AT's local optimality. This work aims to investigate the asymptotic behavior of AT to improve its global performance without losing the local optimality. To this end, a new AT strategy named Adaptive Testing with Gradient Descent method (AT-GD) is proposed. Theoretical analysis indicates that AT-GD, a locally optimal testing strategy, converges to the globally optimal solution as the assessment process proceeds. Simulation and experiments are set up to validate AT-GD's effectiveness and efficiency. Besides, sensitivity analysis of AT-GD is also conducted in this study.
Automatically Finding Performance Problems with Feedback-Directed Learning Software Testing

Mark Grechanik,Chen Fu,Qing Xie

DOI: https://doi.org/10.1109/icse.2012.6227197

2012-01-01

Abstract:A goal of performance testing is to find situations when applications unexpectedly exhibit worsened characteristics for certain combinations of input values. A fundamental question of performance testing is how to select a manageable subset of the input data faster to find performance problems in applications automatically. We offer a novel solution for finding performance problems in applications automatically using black-box software testing. Our solution is an adaptive, feedback-directed learning testing system that learns rules from execution traces of applications and then uses these rules to select test input data automatically for these applications to find more performance problems when compared with exploratory random testing. We have implemented our solution and applied it to a medium-size application at a major insurance company and to an open-source application. Performance problems were found automatically and confirmed by experienced testers and developers.
Adaptive Testing Based on Moment Estimation

Peng Xiao,Yongfeng Yin,Bin Liu,Bo Jiang,Yashwant K. Malaiya

DOI: https://doi.org/10.1109/TSMC.2017.2761767

2020-01-01

Abstract:Adaptive testing (AT) is a software testing approach that uses a feedback mechanism to enhance test effectiveness. Its testing strategy can be adjusted online by using the testing data collected during the software testing process. However, it requires complex parameter estimation which results in excessive computational overhead that may hinder the applicability of AT. In this paper, we propose an approach called AT based on moment estimation (AT-ME) to address this problem. The proposed approach uses moment estimation to serve as the algorithm of parameter estimation, which reduces the complexity of AT-ME. In addition, a dynamic length for testing action is set to limit the number of decisions without influencing the test effectiveness. The proposed approach has been validated on the Siemens test suite, which includes seven real programs. The experiments show that AT-ME can reduce the computational overhead of AT without compromising overall testing efficiency. Results demonstrate that AT-ME is a feasible and effective AT strategy.
Adaptive and Random Partition Software Testing

Junpeng Lv,Hai Hu,Kai-Yuan Cai,Tsong Yueh Chen

DOI: https://doi.org/10.1109/TSMC.2014.2318019

2014-01-01

Abstract:Random testing (RT) and subdomain testing are two major software testing strategies. Their simplicity makes them likely the most efficient testing strategies with respect to the time required for test case selection. However, the disadvantage of RT is its defect detection effectiveness. Adaptive testing (AT) is a feedback-based software testing strategy that has been shown to be more effective than RT and partition testing (PT). However, a major concern in the application of AT is its complexity and computational cost for test case selection. In this paper, we propose a hybrid approach that uses AT and random partition testing (RPT) in an alternating manner. The motivation for this approach is that both strategies are employed such that the underlying computational complexity of AT is reduced by introducing RPT into the testing process without affecting the defect detection effectiveness. A case study with seven real-life subject programs is presented. The experimental results demonstrate that this novel strategy considerably reduces the computational overhead of the original AT strategy but still outperforms the pure RT strategy and PT strategy in terms of the number of test cases used to detect and remove a given number of defects. In addition, a sensitivity analysis is conducted to validate the robustness of our strategy.
A Parallel Implementation Strategy of Adaptive Testing

Hai Hu,Chang-Hai Jiang,Feng Ye,Kai-Yuan Cai,Dazhi Huang,Stephen S. Yau

DOI: https://doi.org/10.1109/COMPSACW.2010.44

2010-01-01

Abstract:Software testing is an important part of software development and can account for more than 50% of the development cost. In order to shorten the testing time and reduce cost, parallel mechanism is introduced to handle multiple testing tasks on the same or different software under tests (SUTs) simultaneously. On the other hand, advanced testing techniques, such as Adaptive Testing (AT) have been proposed to improve the efficiency of traditional random/partition testing. Inspired by the parallel computing technique, we present a parallel implementation of the Adaptive Testing techniques, namely the AT-P strategy. Experiments on the Space program are conducted and data indicates that the AT-P strategy can notably improve the defect detection efficiency of the original adaptive testing technique, whereas providing stable performance enhancement over multiple testing runs.
SOFTWARE TESTING IN THE CONTEXT OF SOFTWARE CYBERNETICS

蔡开元,李永超,景涛,白成刚

DOI: https://doi.org/10.3321/j.issn:1000-6893.2002.05.011

2002-01-01

Abstract:The controlled Markov chains (CMC) approach to software testing wakes the software testing as a control problem. The software under test serves as a controlled object that is modeled as a CMC, and the software testing strategy serves as the corresponding controller. They constitute a closed-loop feedback system, and the testing strategy is designed and optimized in accordance with testing goals, giving a priori in the setting of the theory of CMC. The CMC approach justifies the idea of software cybernetics, which is defined as the interplay between software theory/engineering and control theory/engineering. This paper presents an adaptive approach to the software testing problem and the simulation results show that the adaptive software testing is feasible and is superior to the random testing.

Adaptive software testing with fixed-memory feedback

Adaptive Software Testing with Fixed-Memory Feedback.

A Case Study For Determining Tile Optimal, Size Of Fixed-Memory Feedback In Adaptive Software Testing

An experimental study of adaptive testing for software reliability assessment

An Improved Approach to Adaptive Testing

Adaptive Software Testing in the Context of an Improved Controlled Markov Chain Model

Optimal and Adaptive Testing for Software Reliability Assessment

Adaptive testing of software components

Optimal software testing and adaptive software testing in the context of software cybernetics

A self-adaptive test framework for concurrent programs

Design and Application of a Response-Driven Adaptive Test System

Optimal and Adaptive Testing with Cost Constraints.

On The Online Parameter Estimation Problem In Adaptive Software Testing

Adaptive Partition Testing.

On the Asymptotic Behavior of Adaptive Testing Strategy for Software Reliability Assessment

Automatically Finding Performance Problems with Feedback-Directed Learning Software Testing

Adaptive Testing Based on Moment Estimation

Adaptive and Random Partition Software Testing

A Parallel Implementation Strategy of Adaptive Testing

SOFTWARE TESTING IN THE CONTEXT OF SOFTWARE CYBERNETICS