Completeness of FSM Test Suites Reconsidered

Frits Vaandrager,Paul Fiterău-Broştean,Ivo Melse

2024-10-25

Abstract:A fault domain that has been widely studied in black-box conformance testing is the class of finite state machines (FSMs) with at most $k$ extra states. Numerous methods for generating test suites have been proposed that guarantee fault coverage for this class. These test suites grow exponentially in $k$, so one can only run them for small $k$. But the assumption that $k$ is small is not realistic in practice. As a result, completeness for this fault domain has limited practical significance. As an alternative, we propose (much larger) fault domains that capture the assumption that when bugs in an implementation introduce extra states, these states can be reached via a few (at most $k$) transitions from states reachable via a set $A$ of common scenarios. Preliminary evidence suggests these fault domains, which contain FSMs with an exponential number of extra states (in $k$), are of practical use for testing network protocols. We present a sufficient condition for \emph{$k$-$A$-completeness} of test suites with respect to these fault domains, phrased entirely in terms of properties of their testing tree. Our condition implies $k$-$A$-completeness of two prominent test suite generation algorithms, the Wp and HSI methods. Counterexamples show that three other approaches, the H, SPY and SPYH methods, do not always generate $k$-$A$-complete test suites.

Formal Languages and Automata Theory,Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the completeness problem of finite - state machine (FSM) test suites, especially in black - box conformance testing. Specifically:

1. **Limitations of existing methods**:
- The paper points out that a widely studied class of fault domains in black - box conformance testing is the finite - state machine (FSM) with at most \(k\) extra states. Many methods generate test suites that can guarantee fault coverage for this type of FSM.
- However, the size of these test suites grows exponentially with \(k\), so they can only be run when \(k\) is small. In practical applications, it is not realistic to assume that \(k\) is very small. Therefore, the completeness in this fault domain has limited practical significance.

2. **Proposed new fault domains**:
- As an alternative, the authors propose larger and more practical fault domains. These fault domains assume that when an error in the implementation introduces extra states, these states can be reached through at most \(k\) - step transitions starting from states reachable from a set of common scenarios (represented by set \(A\)).
- These new fault domains include FSMs with an exponential number of extra states (with respect to \(k\)), and preliminary evidence shows that these fault domains have practical uses in network protocol testing.

3. **Completeness conditions of test suites**:
- The authors propose a sufficient condition for determining whether a test suite is complete for the newly proposed fault domain \(k\)-\(A\). This condition is entirely based on the properties of the test tree.
- They prove that two well - known test suite generation algorithms (the Wp method and the HSI method) satisfy this condition, thus ensuring their completeness in the new fault domain.

4. **Comparison with other methods**:
- The authors show by counterexamples that three other methods (the H method, the SPY method, and the SPYH method) do not always generate \(k\)-\(A\) - complete test suites.

In summary, the main objective of the paper is to improve the existing FSM test suite completeness theory to make it more in line with practical application requirements, and to propose new fault domains and corresponding completeness conditions.