Abstract:Verification-aware programming languages such as Dafny and F* provide means to formally specify and prove properties of a program. Although the problem of checking an implementation against a specification can be defined mechanically, there is no algorithmic way of ensuring the correctness of the {\it user-intent formalization for programs}, expressed as a formal specification. This is because intent or requirement is expressed {\it informally} in natural language and the specification is a formal artefact. Despite, the advent of large language models (LLMs) has made tremendous strides bridging the gap between informal intent and formal program implementations recently, driven in large parts by benchmarks and automated metrics for evaluation. Recent work has proposed a framework for evaluating the {\it user-intent formalization} problem for mainstream programming languages~\cite{endres-fse24}. However, such an approach does not readily extend to verification-aware languages that support rich specifications (using quantifiers and ghost variables) that cannot be evaluated through dynamic execution. Previous work also required generating program mutants using LLMs to create the benchmark. We advocate an alternate, perhaps simpler approach of {\it symbolically testing specifications} to provide an intuitive metric for evaluating the quality of specifications for verification-aware languages. We demonstrate that our automated metric agrees closely on a human-labeled dataset of Dafny specifications for the popular MBPP code-generation benchmark, yet demonstrates cases where the human labeling is not perfect. We also outline formal verification challenges that need to be addressed to apply the technique more widely. We believe our work provides a stepping stone to enable the establishment of a benchmark and research agenda for the problem of user-intent formalization for programs.

Staged Specification Logic for Verifying Higher-Order Imperative Programs (Technical Report)

Logical Reasoning for Higher-Order Functions with Local State

Proofs as stateful programs: A first-order logic with abstract Hoare triples, and an interpretation into an imperative language

A Generalized Hybrid Hoare Logic

A Hoare Logic for Domain Specification (Full Version)

Parameterized Dynamic Logic -- Towards A Cyclic Logical Framework for General Program Specification and Verification

Formal Verification of Quantum Algorithms Using Quantum Hoare Logic

A Usage-Aware Sequent Calculus for Differential Dynamic Logic

Scope Logic with Local Reasoning and Pre/Post-State Properties

Evaluating LLM-driven User-Intent Formalization for Verification-Aware Languages

Semantics, Specification Logic, and Hoare Logic of Exact Real Computation

Hybrid: A Definitional Two-Level Approach to Reasoning with Higher-Order Abstract Syntax

From Program Logics to Language Logics

Automated Verification of Higher-Order Probabilistic Programs via a Dependent Refinement Type System

Verifying Java Programs by Theorem Prover HOL

An Applied Quantum Hoare Logic

Specifying Logic Programs in Controlled Natural Language

A Higher-Order Abstract Syntax Approach to the Verified Compilation of Functional Programs

A Theorem Prover for Quantum Hoare Logic and Its Applications

Modular Verification of Linearizability with Non-Fixed Linearization Points