Abstract:Formal verification using proof assistants, such as Coq, is an effective way of improving software quality, but it is expensive. Writing proofs manually requires both significant effort and expertise. Recent research has used machine learning to automatically synthesize proofs, reducing verification effort, but these tools are able to prove only a fraction of the desired software properties. We introduce Cobblestone, a new proof-synthesis approach that improves on the state of the art by taking advantage of partial progress in proof synthesis attempts. Unlike prior tools, Cobblestone can produce multiple unsuccessful proofs using a large language model (LLM), identify the working portions of those proofs, and combine them into a single, successful proof, taking advantage of internal partial progress. We evaluate Cobblestone on two benchmarks of open-source Coq projects, controlling for training data leakage in LLM datasets. Fully automatically, Cobblestone can prove 48% of the theorems, while Proverbot9001, the previous state-of-the-art, learning-based, proof-synthesis tool, can prove 17%. Cobblestone establishes a new state of the art for fully automated proof synthesis tools for Coq. We also evaluate Cobblestone in a setting where it is given external partial proof progress from oracles, serving as proxies for a human proof engineer or another tool. When the theorem is broken down into a set of subgoals and Cobblestone is given a set of relevant lemmas already proven in the project, it can prove up to 58% of the theorems. We qualitatively study the theorems Cobblestone is and is not able to prove to outline potential future research directions to further improve proof synthesis, including developing interactive, semi-automated tools. Our research shows that tools can make better use of partial progress made during proof synthesis to more effectively automate formal verification.

Proof Mining with Dependent Types

Proof mining and probability theory

Statistical Proof Pattern Recognition: Automated or Interactive?

Proof mining for the dual of a Banach space with extensions for uniformly Fréchet differentiable functions

Proof mining in ${\mathbb R}$-trees and hyperbolic spaces

Proof Guidance in PVS with Sequential Pattern Mining.

Dependently Typed Knowledge Graphs

Compositional pre-processing for automated reasoning in dependent type theory

A Practical Formalization of Monadic Equational Reasoning in Dependent-type Theory

Mining Dependence Structures from Statistical Learning Perspective

Learning Structure-Aware Representations of Dependent Types

Premise Selection for Mathematics by Corpus Analysis and Kernel Methods

Automated Theorem Proving via Interacting with Proof Assistants by Dynamic Strategies

Recycling Proof Patterns in Coq: Case Studies

Theorem Proving in Dependently-Typed Higher-Order Logic -- Extended Preprint

A proof-theoretic metatheorem for nonlinear semigroups generated by an accretive operator and applications

Mining Dependent Items

3D-Prover: Diversity Driven Theorem Proving With Determinantal Point Processes

Definitional Functoriality for Dependent (Sub)Types -- Extended version

Coq Modulo Theory - Short Paper

Cobblestone: Iterative Automation for Formal Verification