Abstract:Verification-aware programming languages such as Dafny and F* provide means to formally specify and prove properties of a program. Although the problem of checking an implementation against a specification can be defined mechanically, there is no algorithmic way of ensuring the correctness of the {\it user-intent formalization for programs}, expressed as a formal specification. This is because intent or requirement is expressed {\it informally} in natural language and the specification is a formal artefact. Despite, the advent of large language models (LLMs) has made tremendous strides bridging the gap between informal intent and formal program implementations recently, driven in large parts by benchmarks and automated metrics for evaluation. Recent work has proposed a framework for evaluating the {\it user-intent formalization} problem for mainstream programming languages~\cite{endres-fse24}. However, such an approach does not readily extend to verification-aware languages that support rich specifications (using quantifiers and ghost variables) that cannot be evaluated through dynamic execution. Previous work also required generating program mutants using LLMs to create the benchmark. We advocate an alternate, perhaps simpler approach of {\it symbolically testing specifications} to provide an intuitive metric for evaluating the quality of specifications for verification-aware languages. We demonstrate that our automated metric agrees closely on a human-labeled dataset of Dafny specifications for the popular MBPP code-generation benchmark, yet demonstrates cases where the human labeling is not perfect. We also outline formal verification challenges that need to be addressed to apply the technique more widely. We believe our work provides a stepping stone to enable the establishment of a benchmark and research agenda for the problem of user-intent formalization for programs.

Deductive verification of smart contracts with Dafny

Formal and Executable Semantics of the Ethereum Virtual Machine in Dafny

Case studies of development of verified programs with Dafny for accessibility assessment

dafny-annotator: AI-Assisted Verification of Dafny Programs

Rich Specifications for Ethereum Smart Contract Verification

Foundational Verification of Smart Contracts through Verified Compilation

Who Verifies the Verifiers? A Computer-Checked Implementation of the DPLL Algorithm in Dafny

Towards the verification of a generic interlocking logic: Dafny meets parameterized model checking

Safety Verification of Declarative Smart Contracts

Qafny: A Quantum-Program Verifier

Encoding Induction Proof in Dafny

Well-Behaved (Co)algebraic Semantics of Regular Expressions in Dafny

Leveraging Large Language Models to Boost Dafny's Developers Productivity

Incremental Proof Development in Dafny with Module-Based Induction

Evaluating LLM-driven User-Intent Formalization for Verification-Aware Languages

A refinement-based approach to safe smart contract deployment and evolution

Formally Verifying a Real World Smart Contract

DafnyBench: A Benchmark for Formal Software Verification

Defining the Ethereum Virtual Machine for Interactive Theorem Provers

Verified Development and Deployment of Multiple Interacting Smart Contracts with VeriSolid

Towards AI-Assisted Synthesis of Verified Dafny Methods