Leveraging Large Language Models to Boost Dafny's Developers Productivity

Álvaro Silva,Alexandra Mendes,João F. Ferreira

2024-01-02

Abstract:This research idea paper proposes leveraging Large Language Models (LLMs) to enhance the productivity of Dafny developers. Although the use of verification-aware languages, such as Dafny, has increased considerably in the last decade, these are still not widely adopted. Often the cost of using such languages is too high, due to the level of expertise required from the developers and challenges that they often face when trying to prove a program correct. Even though Dafny automates a lot of the verification process, sometimes there are steps that are too complex for Dafny to perform on its own. One such case is that of missing lemmas, i.e. Dafny is unable to prove a result without being given further help in the form of a theorem that can assist it in the proof of the step.

Software Engineering,Logic in Computer Science,Programming Languages

What problem does this paper attempt to address?

This paper discusses how to enhance the productivity of Dafny programming language developers using Large Language Models (LLMs). Dafny is a language used for software verification, and although it has advantages in ensuring program correctness, its widespread application is limited due to the high level of expertise required from developers.

The paper proposes a new Dafny plugin that leverages LLMs to assist developers in generating relevant theorem suggestions to address complex steps that cannot be automatically completed during the proof process, particularly missing lemmas. Additionally, for lemmas that cannot be automatically proven, the plugin attempts to provide accompanying computational proofs. The researchers also plan to use LLMs to reduce the skill level required for using verification-aware languages, thus increasing their adoption.

The paper focuses on four challenges: lemma and predicate inference, proof inference, automated repair, and code summarization and natural language specifications. In preliminary experiments, they used the GPT-4 model to generate lemma and computational proof suggestions, and the results showed potential despite some errors. However, experiments on proof inference indicated the need for more sophisticated prompt engineering and model fine-tuning to improve the performance of LLMs.

The paper concludes by mentioning future directions, including improving prompt engineering, creating datasets containing relevant Dafny code, and expanding existing datasets such as CloverBench to support more complex examples. Moreover, they plan to integrate these ideas into Dafny's VS Code plugin to enhance the accessibility and efficiency of Dafny for new users.