Abstract:Large language models (LLMs) have shown promise in proving formal theorems using proof assistants such as Lean. However, existing methods are difficult to reproduce or build on, due to private code, data, and large compute requirements. This has created substantial barriers to research on machine learning methods for theorem proving. This paper removes these barriers by introducing LeanDojo: an open-source Lean playground consisting of toolkits, data, models, and benchmarks. LeanDojo extracts data from Lean and enables interaction with the proof environment programmatically. It contains fine-grained annotations of premises in proofs, providing valuable data for premise selection: a key bottleneck in theorem proving. Using this data, we develop ReProver (Retrieval-Augmented Prover): an LLM-based prover augmented with retrieval for selecting premises from a vast math library. It is inexpensive and needs only one GPU week of training. Our retriever leverages LeanDojo's program analysis capability to identify accessible premises and hard negative examples, which makes retrieval much more effective. Furthermore, we construct a new benchmark consisting of 98,734 theorems and proofs extracted from Lean's math library. It features challenging data split requiring the prover to generalize to theorems relying on novel premises that are never used in training. We use this benchmark for training and evaluation, and experimental results demonstrate the effectiveness of ReProver over non-retrieval baselines and GPT-4. We thus provide the first set of open-source LLM-based theorem provers without any proprietary datasets and release it under a permissive MIT license to facilitate further research.

LLMSTEP: LLM proofstep suggestions in Lean

Towards Large Language Models as Copilots for Theorem Proving in Lean

InternLM2.5-StepProver: Advancing Automated Theorem Proving via Expert Iteration on Large-Scale LEAN Problems

Lean-STaR: Learning to Interleave Thinking and Proving

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts

LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

ImProver: Agent-Based Automated Proof Optimization

LeanDojo: Theorem Proving with Retrieval-Augmented Language Models

LeanReasoner: Boosting Complex Logical Reasoning with Lean

LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

Machine-Learned Premise Selection for Lean

DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data

Proof Automation with Large Language Models

LLM4VV: Developing LLM-driven testsuite for compiler validation

Studying and Benchmarking Large Language Models For Log Level Suggestion

$\texttt{LM}^\texttt{2}$: A Simple Society of Language Models Solves Complex Reasoning

LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems

LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations

LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models

Verified Code Transpilation with LLMs

LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error