Abstract:Informal natural language that describes code functionality, such as code comments or function documentation, may contain substantial information about a programs intent. However, there is typically no guarantee that a programs implementation and natural language documentation are aligned. In the case of a conflict, leveraging information in code-adjacent natural language has the potential to enhance fault localization, debugging, and code trustworthiness. In practice, however, this information is often underutilized due to the inherent ambiguity of natural language which makes natural language intent challenging to check programmatically. The emergent abilities of Large Language Models (LLMs) have the potential to facilitate the translation of natural language intent to programmatically checkable assertions. However, it is unclear if LLMs can correctly translate informal natural language specifications into formal specifications that match programmer intent. Additionally, it is unclear if such translation could be useful in practice. In this paper, we describe nl2postcond, the problem of leveraging LLMs for transforming informal natural language to formal method postconditions, expressed as program assertions. We introduce and validate metrics to measure and compare different nl2postcond approaches, using the correctness and discriminative power of generated postconditions. We then use qualitative and quantitative methods to assess the quality of nl2postcond postconditions, finding that they are generally correct and able to discriminate incorrect code. Finally, we find that nl2postcond via LLMs has the potential to be helpful in practice; nl2postcond generated postconditions were able to catch 64 real-world historical bugs from Defects4J.

Improving the Diproche CNL through Autoformalization via Large Language Models

Using Large Language Models for (De-)Formalization and Natural Argumentation Exercises for Beginner's Students

Autoformalization of Game Descriptions using Large Language Models

Proof Automation with Large Language Models

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

Improving Autoformalization using Type Checking

Towards a Mathematics Formalisation Assistant using Large Language Models

Consistent Autoformalization for Constructing Mathematical Libraries

Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency

Number Theory and Axiomatic Geometry in the Diproche System

Multilingual Mathematical Autoformalization

Leveraging Large Language Models to Boost Dafny's Developers Productivity

Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?

Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalization

Towards AI-Assisted Synthesis of Verified Dafny Methods

A New Approach Towards Autoformalization

Towards Large Language Model Aided Program Refinement

Automated Theorem Provers Help Improve Large Language Model Reasoning

Learning from Failures: Translation of Natural Language Requirements into Linear Temporal Logic with Large Language Models

Integration of Large Language Models within Cognitive Architectures for Autonomous Robots

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs