Formally Certified Approximate Model Counting

Yong Kiam Tan,Jiong Yang,Mate Soos,Magnus O. Myreen,Kuldeep S. Meel
2024-06-19
Abstract:Approximate model counting is the task of approximating the number of solutions to an input Boolean formula. The state-of-the-art approximate model counter for formulas in conjunctive normal form (CNF), ApproxMC, provides a scalable means of obtaining model counts with probably approximately correct (PAC)-style guarantees. Nevertheless, the validity of ApproxMC's approximation relies on a careful theoretical analysis of its randomized algorithm and the correctness of its highly optimized implementation, especially the latter's stateful interactions with an incremental CNF satisfiability solver capable of natively handling parity (XOR) constraints. We present the first certification framework for approximate model counting with formally verified guarantees on the quality of its output approximation. Our approach combines: (i) a static, once-off, formal proof of the algorithm's PAC guarantee in the Isabelle/HOL proof assistant; and (ii) dynamic, per-run, verification of ApproxMC's calls to an external CNF-XOR solver using proof certificates. We detail our general approach to establish a rigorous connection between these two parts of the verification, including our blueprint for turning the formalized, randomized algorithm into a verified proof checker, and our design of proof certificates for both ApproxMC and its internal CNF-XOR solving steps. Experimentally, we show that certificate generation adds little overhead to an approximate counter implementation, and that our certificate checker is able to fully certify $84.7\%$ of instances with generated certificates when given the same time and memory limits as the counter.
Logic in Computer Science,Artificial Intelligence
What problem does this paper attempt to address?
### Problems Addressed by the Paper The paper aims to address the verification problem in approximate model counting. Specifically, the task of approximate model counting is to estimate the number of solutions for a given Boolean formula. Although the state-of-the-art approximate model counter ApproxMC provides a scalable method to obtain model counts with Probably Approximately Correct (PAC) guarantees, the validity of its results relies on the theoretical analysis of its randomized algorithm and the correctness of its implementation, particularly its interaction with the incremental CNF satisfiability solver. ### Main Contributions 1. **Formal Verification Framework**: The authors propose the first approximate model counting certification framework with formal verification guarantees. This framework combines: - A static one-time formal proof, proving the algorithm's PAC guarantee, completed in the Isabelle/HOL proof assistant. - A dynamic per-run verification, validating ApproxMC's calls to the external CNF-XOR solver through proof certificates. 2. **Certificate Format and Checker Implementation**: The authors refine the abstract specification into a concrete certificate format and implement a certificate checker. 3. **Tool Updates**: Various tools were updated to enable a formally verified proof-checking pipeline supporting CNF-XOR unsatisfiability proofs. 4. **Experimental Evaluation**: The practical utility of the framework is demonstrated through extensive model counting benchmarks. ### Key Techniques - **Formal Analysis of Randomized Algorithms**: The PAC guarantee of ApproxMC is formalized in Isabelle/HOL. - **Certificate Generation and Checking**: A certificate format is designed, containing only solver call results crucial to ApproxMC's correctness. - **CNF-XOR Unsatisfiability Checking**: The existing solving and proof-checking pipeline is adapted to support proof certificates for CNF-XOR unsatisfiability. ### Experimental Results Experiments show that certificate generation incurs minimal overhead on the approximate counter implementation, and under the same time and memory constraints, the certificate checker can fully verify 84.7% of instances. ### Impact - **Enhanced User Trust**: Formal verification enhances user trust in the results of approximate model counting. - **Error Detection and Fixing**: During the modification of the underlying solver CryptoMiniSat, several errors were discovered and fixed, including bugs in the XOR operations, minor flaws in theoretical analysis, and implementation deviations. ### Related Work - **Certificates for Deterministic Methods**: Previous certificate methods mainly focused on deterministic model counting. - **Formalization of Randomized Algorithms**: Various randomized algorithms, such as randomized quicksort and randomized binary tree data structures, have been formalized in Isabelle/HOL. - **Proof Checkers for Formal Verification**: Multiple proof checkers for formal verification have been developed to check CNF unsatisfiability proofs in the SAT community. ### Conclusion The paper proposes a hybrid approach combining static formal proofs and dynamic certificate checking to verify the randomized approximate model counting system ApproxMC. This framework not only enhances user trust in the results but also identifies and fixes multiple errors in existing tools.