Abstract:Formal verification using proof assistants, such as Coq, is an effective way of improving software quality, but it is expensive. Writing proofs manually requires both significant effort and expertise. Recent research has used machine learning to automatically synthesize proofs, reducing verification effort, but these tools are able to prove only a fraction of the desired software properties. We introduce Cobblestone, a new proof-synthesis approach that improves on the state of the art by taking advantage of partial progress in proof synthesis attempts. Unlike prior tools, Cobblestone can produce multiple unsuccessful proofs using a large language model (LLM), identify the working portions of those proofs, and combine them into a single, successful proof, taking advantage of internal partial progress. We evaluate Cobblestone on two benchmarks of open-source Coq projects, controlling for training data leakage in LLM datasets. Fully automatically, Cobblestone can prove 48% of the theorems, while Proverbot9001, the previous state-of-the-art, learning-based, proof-synthesis tool, can prove 17%. Cobblestone establishes a new state of the art for fully automated proof synthesis tools for Coq. We also evaluate Cobblestone in a setting where it is given external partial proof progress from oracles, serving as proxies for a human proof engineer or another tool. When the theorem is broken down into a set of subgoals and Cobblestone is given a set of relevant lemmas already proven in the project, it can prove up to 58% of the theorems. We qualitatively study the theorems Cobblestone is and is not able to prove to outline potential future research directions to further improve proof synthesis, including developing interactive, semi-automated tools. Our research shows that tools can make better use of partial progress made during proof synthesis to more effectively automate formal verification.

BliStrTune: Hierarchical Invention of Theorem Proving Strategies

Solving Hard Mizar Problems with Instantiation and Strategy Invention

BAIT: Benchmarking (Embedding) Architectures for Interactive Theorem-Proving

An evolutionary/heuristic-based proof searching framework for interactive theorem prover

Automated Theorem Proving in Intuitionistic Propositional Logic by Deep Reinforcement Learning

MaLeS: A Framework for Automatic Tuning of Automated Theorem Provers

Proof searching and prediction in HOL4 with evolutionary/heuristic and deep learning techniques

Lean-STaR: Learning to Interleave Thinking and Proving

Towards an Intelligent Tutor for Mathematical Proofs

Hammering Mizar by Learning Clause Guidance

Learning to Reason with HOL4 tactics

A Synthesis of the Procedural and Declarative Styles of Interactive Theorem Proving

Top-down Automated Theorem Proving (Notes for Sir Timothy)

Beyond Notations: Hygienic Macro Expansion for Theorem Proving Languages

Proving Theorems Recursively

Strips: A new approach to the application of theorem proving to problem solving

Proving Theorems using Incremental Learning and Hindsight Experience Replay

Automated Strategy Invention for Confluence of Term Rewrite Systems

From {Solution Synthesis} to {Student Attempt Synthesis} for Block-Based Visual Programming Tasks

3D-Prover: Diversity Driven Theorem Proving With Determinantal Point Processes

Cobblestone: Iterative Automation for Formal Verification