Abstract:Formal verification using proof assistants, such as Coq, is an effective way of improving software quality, but it is expensive. Writing proofs manually requires both significant effort and expertise. Recent research has used machine learning to automatically synthesize proofs, reducing verification effort, but these tools are able to prove only a fraction of the desired software properties. We introduce Cobblestone, a new proof-synthesis approach that improves on the state of the art by taking advantage of partial progress in proof synthesis attempts. Unlike prior tools, Cobblestone can produce multiple unsuccessful proofs using a large language model (LLM), identify the working portions of those proofs, and combine them into a single, successful proof, taking advantage of internal partial progress. We evaluate Cobblestone on two benchmarks of open-source Coq projects, controlling for training data leakage in LLM datasets. Fully automatically, Cobblestone can prove 48% of the theorems, while Proverbot9001, the previous state-of-the-art, learning-based, proof-synthesis tool, can prove 17%. Cobblestone establishes a new state of the art for fully automated proof synthesis tools for Coq. We also evaluate Cobblestone in a setting where it is given external partial proof progress from oracles, serving as proxies for a human proof engineer or another tool. When the theorem is broken down into a set of subgoals and Cobblestone is given a set of relevant lemmas already proven in the project, it can prove up to 58% of the theorems. We qualitatively study the theorems Cobblestone is and is not able to prove to outline potential future research directions to further improve proof synthesis, including developing interactive, semi-automated tools. Our research shows that tools can make better use of partial progress made during proof synthesis to more effectively automate formal verification.

Advances of Proof Scores in CafeOBJ

Transforming ASN.1 Specifications into CafeOBJ to assist with Property Checking

Automating Verification of Event-B Models

Your Proof Fails? Testing Helps to Find the Reason

Formal Specification and Analysis of Autonomous Systems under Partial Compliance

Certifying Zero-Knowledge Circuits with Refinement Types

Towards a Scalable Proof Engine: A Performant Prototype Rewriting Primitive for Coq

A SAT Solver and Computer Algebra Attack on the Minimum Kochen-Specker Problem

Quantitative Assurance and Synthesis of Controllers from Activity Diagrams

A Formal Verification Method for the SOPC Software

AC4: Algebraic Computation Checker for Circuit Constraints in ZKPs

Formal Specification and Analysis of Partitioning Operating Systems by Integrating Ontology and Refinement.

Quantifying Software Correctness by Combining Architecture Modeling and Formal Program Analysis

An Automatic Test Coverage Analysis For Systemc Description Using Aspect-Oriented Programming

Natural Language Specifications in Proof Assistants

Axioms And Abstract Predicates On Interfaces In Specifying/Verifying Oo Components

Cobblestone: Iterative Automation for Formal Verification

Validation Obligations: A Novel Approach to Check Compliance between Requirements and their Formal Specification

Diagnosis via Proofs of Unsatisfiability for First-Order Logic with Relational Objects

Logical analysis and contradiction detection in high-level requirements during the review process using sat-solver

Scenario-Based Proofs for Concurrent Objects [Extended Version]