Abstract:Owing to the pervasiveness of software in our modern lives, software systems have evolved to be highly configurable. Combinatorial testing has emerged as a dominant paradigm for testing highly configurable systems. Often constraints are employed to define the environments where a given system is expected to work. Therefore, there has been a sustained interest in designing constraint-based test suite generation techniques. A significant goal of test suite generation techniques is to achieve t-wise coverage for higher values of t. Therefore, designing scalable techniques that can estimate t-wise coverage for a given set of tests and/or the estimation of maximum achievable t-wise coverage under a given set of constraints is of crucial importance. The existing estimation techniques face significant scalability hurdles. We designed scalable algorithms with mathematical guarantees to estimate (i) t-wise coverage for a given set of tests, and (ii) maximum t-wise coverage for a given set of constraints. In particular, ApproxCov takes in a test set U and returns an estimate of the t-wise coverage of U that is guaranteed to be within (1±ε)-factor of the ground truth with probability at least 1−δ for a given tolerance parameter ε and a confidence parameter δ. A scalable framework ApproxMaxCov for a given formula F outputs an approximation which is guaranteed to be within (1±ε) factor of the maximum achievable t-wise coverage under F, with probability ≥1−δ for a given tolerance parameter ε and a confidence parameter δ. Our comprehensive evaluation demonstrates that ApproxCov and ApproxMaxCov can handle benchmarks that are beyond the reach of current state-of-the-art approaches. In this paper we present proofs of correctness of ApproxCov, ApproxMaxCov, and of their generalizations. We show how the algorithms can improve the scalability of a test suite generator while maintaining its effectiveness. In addition, we compare several test suite generators on different feature combination sizes t.

Cross-coverage testing of functionally equivalent programs

Identify and Update Test Cases when Production Code Changes: A Transformer-Based Approach

Empirical Evaluation of Test Coverage for Functional Programs.

CoverUp: Coverage-Guided LLM-Based Test Generation

Guided test generation for coverage criteria

Code Coverage-Based Failure Proximity Without Test Oracles.

Extended Covering Arrays for Sequence Coverage.

Heterogeneous Testing for Coverage Profilers Empowered with Debugging Support

A Scalable t-Wise Coverage Estimator: Algorithms and Applications

PEQtest: Testing Functional Equivalence

The Study of Pairwise Condition Covering Test Technology

Top-down: A better strategy for incremental covering array generation

Test Coverage Analysis Based on Program Slicing

Equivalence testing for multiple groups

Hunting for bugs in code coverage tools via randomized differential testing

Coverage Explorer: Coverage-guided Test Generation for Cyber Physical Systems

Covana: Precise Identification Of Problems In Pex

On-line tracing of XACML-based policy coverage criteria

Hybrid Multi-level Crossover for Unit Test Case Generation

Defining and Measuring Policy Coverage in Testing Access Control Policies.