Mica: Automated Differential Testing for OCaml Modules

Ernest Ng,Harrison Goldstein,Benjamin C. Pierce

2024-08-27

Abstract:Suppose we are given two OCaml modules implementing the same signature. How do we check that they are observationally equivalent -- that is, that they behave the same on all inputs? One established technique is to use a property-based testing (PBT) tool such as QuickCheck. Currently, however, this can require significant amounts of boilerplate code and ad-hoc test harnesses. To address this issue, we present Mica, an automated tool for testing observational equivalence of OCaml modules. Mica is implemented as a PPX compiler extension, allowing users to supply minimal annotations to a module signature. These annotations guide Mica to automatically derive specialized PBT code that checks observational equivalence. We discuss the design of Mica and demonstrate its efficacy as a testing tool on various modules taken from real-world OCaml libraries.

Programming Languages,Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to check whether two OCaml modules with the same signature are observationally equivalent on all inputs (i.e., whether their behaviors are the same). Current methods usually rely on Property - Based Testing (PBT) tools, such as QuickCheck. However, these methods often require a large amount of boilerplate code and custom test frameworks, increasing the burden on users.

To solve this problem, the author proposes Mica, an automated tool for testing the equivalence of OCaml modules. Mica simplifies this process in the following ways:

1. **Automated generation of test code**: Mica is implemented as a PPX compiler extension. Users only need to add a few annotations (such as `[@@deriving mica]`) in the module signature, and Mica will automatically generate specialized PBT code to check observational equivalence.
2. **Symbolic expression generation**: Mica generates a recursive QuickCheck generator `gen_expr`, which is parameterized according to the type of the required expression, ensuring that only type - correct expressions are generated.
3. **Interpretation of symbolic expressions**: Mica also generates an interpreter that can interpret symbolic expressions into concrete values and execute them on a given concrete module.
4. **Differential testing framework**: Mica provides a differential testing framework `TestHarness`, and users can test the equivalence of two modules by instantiating this framework.

In addition, Mica is integrated with the Tyche tool to visualize the behavior of property - based test generators, helping users better understand the effectiveness of the tests. Through multiple case studies, Mica has demonstrated its ability to detect manually inserted errors and errors in student assignments, proving its effectiveness in practical applications.

In summary, the main goal of Mica is to reduce the amount of work for users to write and maintain complex test code, while increasing the degree of automation and accuracy of testing.