A Review of Pair-wise Testing

Jimi Sanchez

DOI: https://doi.org/10.48550/arXiv.1606.00288

2016-06-01

Software Engineering

Abstract:In software testing, the large size of the input domain makes exhaustively testing the inputs a daunting and often impossible task. Pair-wise testing is a popular approach to combinatorial testing problems. This paper reviews Pair-wise testing and its history, strengths, weaknesses, and tools for generating test cases.

What problem does this paper attempt to address?

A Software Debugging Method Based on Pairwise Testing

L Shi,CH Nie,BW Xu

DOI: https://doi.org/10.1007/11428862_179

2005-01-01

Abstract:Pairwise testing is one of very practical and effective testing methods for various types of software systems. This paper proposes a novel debugging method based on pairwise testing. By analyzing the test cases and retesting with some complementary test cases, the method can narrow the factors that cause the errors in a very small range. So it can provide a very efficient and valuable guidance for the software testing and debugging.
Can Pairwise Testing Perform Comparably to Manually Handcrafted Testing Carried Out by Industrial Engineers?

Peter Charbachi,Linus Eklund,Eduard Enoiu

DOI: https://doi.org/10.48550/arXiv.1706.01636

2017-06-06

Abstract:Testing is an important activity in engineering of industrial software. For such software, testing is usually performed manually by handcrafting test suites based on specific design techniques and domain-specific experience. To support developers in testing, different approaches for producing good test suites have been proposed. In the last couple of years combinatorial testing has been explored with the goal of automatically combining the input values of the software based on a certain strategy. Pairwise testing is a combinatorial technique used to generate test suites by varying the values of each pair of input parameters to a system until all possible combinations of those parameters are created. There is some evidence suggesting that these kinds of techniques are efficient and relatively good at detecting software faults. Unfortunately, there is little experimental evidence on the comparison of these combinatorial testing techniques with, what is perceived as, rigorous manually handcrafted testing. In this study we compare pairwise test suites with test suites created manually by engineers for 45 industrial programs. The test suites were evaluated in terms of fault detection, code coverage and number of tests. The results of this study show that pairwise testing, while useful for achieving high code coverage and fault detection for the majority of the programs, is almost as effective in terms of fault detection as manual testing. The results also suggest that pairwise testing is just as good as manual testing at fault detection for 64% of the programs.

Software Engineering
Applying Particle Swarm Optimization to Pairwise Testing

Xiang Chen,Qing Gu,Jingxian Qi,Daoxu Chen

DOI: https://doi.org/10.1109/COMPSAC.2010.17

2010-01-01

Abstract:Combinatorial testing (also called interaction testing) is an effective specification-based test input generation technique. By now most of research work in combinatorial testing aims to propose novel approaches trying to generate test suites with minimum size that still cover all the pairwise, triple, or n-way combinations of factors. Since the difficulty of solving this problem is demonstrated to be NP-hard, existing approaches have been designed to generate optimal or near optimal combinatorial test suites in polynomial time. In this paper, we try to apply particle swarm optimization (PSO), a kind of meta-heuristic search technique, to pairwise testing (i.e. a special case of combinatorial testing aiming to cover all the pairwise combinations). To systematically build pairwise test suites, we propose two different PSO based algorithms. One algorithm is based on one-test-at-a-time strategy and the other is based on IPO-like strategy. In these two different algorithms, we use PSO to complete the construction of a single test. To successfully apply PSO to cover more uncovered pairwise combinations in this construction process, we provide a detailed description on how to formulate the search space, define the fitness function and set some heuristic settings. To verify the effectiveness of our approach, we implement these algorithms and choose some typical inputs. In our empirical study, we analyze the impact factors of our approach and compare our approach to other well-known approaches. Final empirical results show the effectiveness and efficiency of our approach.
Design and Implement of a Tool for Pairwise Testing Data Generation

ZHOU Xiao-Yu,NIE Chang-Hai,XU Bao-Wen,CHEN Huo-Wang

2005-01-01

Computer Science

Abstract:Pairwise testing is practical and effective for various types of software systems,and it is a key problem that the test data generation method for the pairwise testing. This paper introduces the design and implement of a tool for the pairwise test data generation,which is based on the foundation of the former and our recent research results. The experiments in using this tool show that our tool has good properties and merits compared with the related tool.
Pairwise Test Data Generation Based on Solution Space Tree

Liang Shi,Nie Chang

2006-01-01

Chinese Journal of Computers

Abstract:Pairwise testing is practical and effective for various types of software systems,and it is a key problem to generate test data for the pairwise coverage criterion.This paper represents all feasible test data as a solution space tree based on the combinatorial testing model,and suggests a backtrack approach to search a subset of paths in the tree for pairwise testing data generation.Then some test data may be added into the test data suite by a greedy algorithm to cover all the uncovered pairs.The authors have implemented a test generation tool based on the approach,and the empirical results show that the approach has some merits compared with the related works.
Improved Combinatorial Algorithms Test for Pairwise Testing Used for Testing Data Generation in Big Data Applications

Deepa Gupta,Lavanya Sharma

DOI: https://doi.org/10.1201/9781003140351-9

IF: 14.4

2021-10-05

Artificial Intelligence

Abstract:These days a huge amount of data is generated, which is effectively tackled by various big data applications. However, it takes a lot of time to process these vast volumes of data. Hence, while developing big data applications, we need to build effective as well as smaller datasets. In this chapter we present an improved algorithm of pair-wise testing for this purpose. This improved algorithm of pairwise testing can be effectively used in big data applications. Pairwise testing is expensive and time consuming. In order to deal with the problems that are present in pairwise testing and exploiting the abundance of data that is presents, artificial intelligence (AI) and machine learning can be used to make the process of testing easier. These tests are conducted by using the ability of AI to create more advanced tests: the algorithm finds the pattern in the data and creates a comprehensive data model of observed parameters.

computer science, artificial intelligence
A New Pairwise Covering Test Data Generation Algorithm for the System with Many 2-Level Factors

Changhai Nie,XU Bao-Wen,SHI Liang

DOI: https://doi.org/10.3321/j.issn:0254-4164.2006.06.001

2006-01-01

Chinese Journal of Computers

Abstract:Pairwise testing is very practical and effective.Much research has been done in the test data generation algorithm.The authors first study pairwise testing data generated for SM2(System with Many 2-level factors) with the existed methods,then present a new algorithm by combinatorial analysis to avoid the shortages that the existed methods are not good enough for SM2.This algorithm can generate the most effective test suite with smaller size.The authors use the result to test the Linux code and design the test plan for the software configuration testing.It shows the test data generated by the algorithm has high code coverage and fault-detection ability.
Heterogeneous and Homogenous Pairs in Pair Programming: an Empirical Analysis

Mujeeb-u-Rehman, M.,Xiaohu Yang,Jinxiang Dong,Memon Abdul Ghafoor

DOI: https://doi.org/10.1109/ccece.2005.1557172

2005-01-01

Abstract:Extreme programming (XP) is causing a lot of hype currently. Pair programming is one of its practices, where two programmers work collaboratively at one computer on the same design, algorithm, code, or test. The purpose of this technique is to produce higher quality code in less time. The collaborative working of two brains is better than one as long as both are actively trying to accomplish the same goal. It depends on many things, such as creativity, productivity and brainstorming which are the essential factors of the programming activity. The existence of these factors can vary in homogenous and heterogeneous pairs, so the functional brainstorming capability of different pairs during programming may vary in terms of quality, accuracy and speed. This paper presents the empirical analysis between the homogenous and heterogeneous pairs in terms of efficiency, accuracy, quality, quantity of bugs, brainstorming performance and creativity potential during programming activity
A Pairwise T-Way Test Suite Generation Strategy Using Gravitational Search Algorithm

Khin Maung Htay,Rozmie Razif Othman,Amiza Amir,Hasneeza Liza Zakaria,Nuraminah Ramli

DOI: https://doi.org/10.1109/ICAICST53116.2021.9497823

2021-07-29

Abstract:Software faults are commonly occurred due to interactions between one or more input parameters in complex software systems. Software test design techniques can be implemented to ensure the quality of the developed software. Exhaustive testing tests all possible test configurations; however, it is infeasible considering time and resource constraints. Pairwise t-way testing is a sampling strategy that focuses on testing every pair of parameter combination, effectively reducing the generated test size as opposed to testing exhaustively. In this paper, we propose a new pairwise t-way strategy called Pairwise Gravitational Search Algorithm Strategy (PGSAS). PGSAS utilizes Gravitational Search Algorithm (GSA) for generating optimal pairwise test suites. The performance of PGSAS is benchmarked against existing t-way strategies in terms of test suite size. Preliminary results showcase that PGSAS provides competitive results in most configurations and outshines other strategies in some cases.

Software Engineering
Framework of Particle Swarm Optimization Based Pairwise Testing

CHEN Xiang,GU Qing,WANG Zi-Yuan,CHEN Dao-Xu

DOI: https://doi.org/10.3724/sp.j.1001.2011.03973

2011-01-01

Abstract:This paper proposes a framework of particle swarm optimization (PSO) based pairwise testing.To systematically build pairwise test suites, two different PSO based strategies are proposed.One strategy takes on a one-test-at-a-time approach and the other takes on an IPO-like approach.In these two different strategies, PSO is used to complete the construction of a single test and research on how to formulate the search space, define the fitness function, and set some heuristic settings.To verify the effectiveness of this approach, these algorithms are implemented and some typical instances have been chosen.In this empirical study, the paper analyzes the impact factors of this framework and compares this approach to other well-known approaches in test suite size and generation time.Final empirical results show the competitiveness of this approach.
Generating Pairwise Combinatorial Interaction Test Suites Using Single Objective Dragonfly Optimisation Algorithm

Bestoun S. Ahmed

DOI: https://doi.org/10.17656/jzs.10586

2019-04-10

Abstract:Combinatorial interaction testing has been addressed as an effective software testing technique recently. It shows its ability to reduce the number of test cases that have to be considered for software-under-test by taking the combinations of parameters as an interaction of input. This combination could be considered as input-configuration of different software families. Pairwise combinatorial test suite takes the interaction of two input parameters into consideration instead of many parameter interactions. Evidence showed that this test suite could detect most of the faults in the software-under-test as compared to higher interactions. This paper presents a new technique to generate pairwise combinatorial test suites. Also, Dragon Fly (DF), a new swarm intelligent optimization algorithm, is assessed. The design and adaptation of the algorithm are addresses in the paper in detail. The algorithm is evaluated extensively through different experiments and benchmarks. The evaluation shows the efficiency of the proposed technique for test suite generation and the usefulness of DF optimization algorithm for future investigations.

Software Engineering
A Comparative Review of Methods for Comparing Means Using Partially Paired Data

Beibei Guo,Ying Yuan

DOI: https://doi.org/10.1177/0962280215577111

IF: 2.494

2015-01-01

Statistical Methods in Medical Research

Abstract:In medical experiments with the objective of testing the equality of two means, data are often partially paired by design or because of missing data. The partially paired data represent a combination of paired and unpaired observations. In this article, we review and compare nine methods for analyzing partially paired data, including the two-sample t-test, paired t-test, corrected z-test, weighted t-test, pooled t-test, optimal pooled t-test, multiple imputation method, mixed model approach, and the test based on a modified maximum likelihood estimate. We compare the performance of these methods through extensive simulation studies that cover a wide range of scenarios with different effect sizes, sample sizes, and correlations between the paired variables, as well as true underlying distributions. The simulation results suggest that when the sample size is moderate, the test based on the modified maximum likelihood estimator is generally superior to the other approaches when the data is normally distributed and the optimal pooled t-test performs the best when the data is not normally distributed, with well-controlled type I error rates and high statistical power; when the sample size is small, the optimal pooled t-test is to be recommended when both variables have missing data and the paired t-test is to be recommended when only one variable has missing data.
Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests.

Anjiang Wei,Pu Yi,Tao Xie,Darko Marinov,Wing Lam

DOI: https://doi.org/10.1007/978-3-030-72016-2_15

2021-01-01

Abstract:AbstractSoftware developers frequently check their code changes by running a set of tests against their code. Tests that can nondeterministically pass or fail when run on the same code version are called flaky tests. These tests are a major problem because they can mislead developers to debug their recent code changes when the failures are unrelated to these changes. One prominent category of flaky tests is order-dependent (OD) tests, which can deterministically pass or fail depending on the order in which the set of tests are run. By detecting OD tests in advance, developers can fix these tests before they change their code. Due to the high cost required to explore all possible orders (n! permutations for n tests), prior work has developed tools that randomize orders to detect OD tests. Experiments have shown that randomization can detect many OD tests, and that most OD tests depend on just one other test to fail. However, there was no analysis of the probability that randomized orders detect OD tests. In this paper, we present the first such analysis and also present a simple change for sampling random test orders to increase the probability. We finally present a novel algorithm to systematically explore all consecutive pairs of tests, guaranteeing to detect all OD tests that depend on one other test, while running substantially fewer orders and tests than simply running all test pairs.
Combinatorial Testing and Model‐based Testing

Robert M. Hierons,Tao Xie

DOI: https://doi.org/10.1002/stvr.1810

2022-01-01

Software Testing Verification and Reliability

Abstract:This issue contains two papers. The first paper focuses on combinatorial testing and the second one focuses on model-based testing. The first paper, ‘Combinatorial methods for testing Internet of Things smart home systems’ by Bernhard Garn, Dominik-Philip Schreiber, Dimitris E. Simos, Rick Kuhn, Jeff Voas, and Raghu Kacker, presents an approach for applying combinatorial testing (CT) to the internal configuration and functionality of Internet of Things (IoT) home automation hub systems. The authors first create an input parameter model of an IoT home automation hub system for use with test generation strategies of combinatorial testing and then propose an automated test execution framework and two test oracles for evaluation purposes. The proposed approach makes use of the appropriately formulated model of the hub and generates test sets derived from this model satisfying certain combinatorial coverage conditions. The authors conduct an evaluation of the proposed approach on a real-world IoT system. The evaluation results show that the proposed approach reveals multiple errors in the devices under test, and all approaches under comparison perform nearly equally well (recommended by W. K. Chan). The second paper, ‘Effective grey-box testing with partial FSM models’ by Robert Sachtleben and Jan Peleska, explores the problem of testing from a finite state machine (FSM) and considers the scenario in which an input can be enabled in some states and disabled in other states. There is already a body of work on testing from FSMs in which inputs are not always defined (partial FSMs), but such work typically allows the system under test (SUT) to be such that some inputs are defined in a state of the SUT but are not defined in the corresponding state of the specification FSM (the SUT can be ‘more’ defined). The paper introduces a conformance relation, called strong reduction, that requires that exactly the same inputs are defined in the specification and the SUT. A new test generation technique is given for strong reduction, with this returning test suites that are complete: a test suite is guaranteed to fail if the SUT is faulty and also satisfies certain conditions that place an upper bound on the number of states of the SUT. The overall approach also requires that the tester can determine which inputs are enabled in the current state of the SUT and so testing is grey-box (recommended by Helene Waeselynck).
A survey of combinatorial testing

Changhai Nie,Hareton Leung

DOI: https://doi.org/10.1145/1883612.1883618

2011-01-01

Abstract:Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.
A Method Combining Review and Testing for Verifying Software Systems

Yuting Chen,Shaoying Liu,W. Eric Wong

DOI: https://doi.org/10.1109/BMEI.2008.332

2008-01-01

Abstract:The application of specification-based program verification techniques (e.g., black-box testing, formal proof) faces strong challenges in practice when the gap between the structure of a specification and that of its program is large. This paper describes a method combining program review and testing to address these challenges. The essential idea of the combination is first to use a view-based program review approach to maintain consistency between the specification structure and the program structure, and then to use a view-based testing approach to dynamically check whether the program satisfies the functional requirements defined in the specification. Views are used to serve as the basis for reviewing structural consistency and for testing functional consistency.
Generating combinatorial test suite for interaction relationship.

Ziyuan Wang,Changhai Nie,Baowen Xu

DOI: https://doi.org/10.1145/1295074.1295085

2007-01-01

Abstract:ABSTRACTCombinatorial testing could detect the faults triggered by the interactions among factors in software. But in many cases, the pair-wise, N-way and even the variable strength combinatorial testing may lead test suite redundancy and fault detect ability decreasing, because these methods do not make sufficient consideration on the actual factors interaction. In this paper, a new interaction relationship based combinatorial testing model was proposed to cover the actual factor interactions in software by extending the conventional combinatorial testing model and IO relationship testing model. The new method may be more effectively than existed combinatorial testing methods without decrease of the fault detect ability. Furthermore, two test suite generation algorithms for interaction relationship based combinatorial testing were also presented. Finally, we compared our algorithms with some similar test generation algorithms in IO relationship testing model, and the experience result showed the advantage of our algorithms.
Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization

Xiang Chen,Qing Gu,Xin Zhang,Daoxu Chen

DOI: https://doi.org/10.1109/QSIC.2009.52

2009-01-01

Abstract:Interaction testing offers a stable cost-benefit ratio in identifying faults. But in many testing scenarios, the entire test suite cannot be fully executed due to limited time or cost. In these situations, it is essential to take the importance of interactions into account and prioritize these tests. To tackle this issue, the biased covering array is proposed and the Weighted Density Algorithm (WDA) is developed. To find a better solution, in this paper we adopt ant colony optimization (ACO) to build this prioritized pairwise interaction test suite (PITS). In our research, we propose four concrete test generation algorithms based on Ant System, Ant System with Elitist, Ant Colony System and Max-Min Ant System respectively. We also implement these algorithms and apply them to two typical inputs and report experimental results. The results show the effectiveness of these algorithms.
Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite.

Wang Ziyuan,Xu Baowen,Nie Changhai

DOI: https://doi.org/10.1109/qsic.2008.52

2008-01-01

Abstract:Combinatorial testing is a practical software testing approach that has been widely used in practice. Most research and applications of such approach focus on N-way combinatorial testing that provides a minimum coverage of all N-way interactions among factor. However, the strengths of different interactions may not be a fixed integer N, but a variable. Therefore, variable strength combinatorial testing approach is necessary in applications. Existing variable strength combinatorial testing, which allows some interactions have a higher strength than others, has a limitation that such higher-strength interactions must be disjoint. To avoid such a limitation, an improved variable strength combinatorial testing approach, which makes a more sufficient consideration on actual interaction relationship, is proposed in this article. Furthermore, two greedy heuristic algorithms are also proposed to generate combinatorial test suite. Compared to some existing algorithms and tools, the proposed algorithms have advantages on both the execution effectiveness and the optimality of generated test suite. Experimental results can prove such advantages.
The Study of Pairwise Condition Covering Test Technology

Shi Zhu,Qing Ding,Ming Jiang

DOI: https://doi.org/10.1109/icise.2010.5689928

2010-01-01

Abstract:Multiple Condition Coverage (MCC) is a very strong test criterion in the white-box test, but as it requires much more test cases, it is not useful in practices. We now propose a new method: Pairwise Condition Covering test, which simplified MCC, but inherited the advantage of MCC, meanwhile it needs only a few numbers of test cases. Compare to the widely used Modified Condition/Decision Coverage (MC/DC), it will use less test cases when there are quite a lot of conditions; besides it can discover the deficiencies which MC / DC could not identify.

A Review of Pair-wise Testing

A Software Debugging Method Based on Pairwise Testing

Can Pairwise Testing Perform Comparably to Manually Handcrafted Testing Carried Out by Industrial Engineers?

Applying Particle Swarm Optimization to Pairwise Testing

Design and Implement of a Tool for Pairwise Testing Data Generation

Pairwise Test Data Generation Based on Solution Space Tree

Improved Combinatorial Algorithms Test for Pairwise Testing Used for Testing Data Generation in Big Data Applications

A New Pairwise Covering Test Data Generation Algorithm for the System with Many 2-Level Factors

Heterogeneous and Homogenous Pairs in Pair Programming: an Empirical Analysis

A Pairwise T-Way Test Suite Generation Strategy Using Gravitational Search Algorithm

Framework of Particle Swarm Optimization Based Pairwise Testing

Generating Pairwise Combinatorial Interaction Test Suites Using Single Objective Dragonfly Optimisation Algorithm

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

Probabilistic and Systematic Coverage of Consecutive Test-Method Pairs for Detecting Order-Dependent Flaky Tests.

Combinatorial Testing and Model‐based Testing

A survey of combinatorial testing

A Method Combining Review and Testing for Verifying Software Systems

Generating combinatorial test suite for interaction relationship.

Building Prioritized Pairwise Interaction Test Suites with Ant Colony Optimization

Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite.

The Study of Pairwise Condition Covering Test Technology