Guiding Enumerative Program Synthesis with Large Language Models

Yixuan Li,Julian Parsert,Elizabeth Polgreen

2024-03-07

Abstract:Pre-trained Large Language Models (LLMs) are beginning to dominate the discourse around automatic code generation with natural language specifications. In contrast, the best-performing synthesizers in the domain of formal synthesis with precise logical specifications are still based on enumerative algorithms. In this paper, we evaluate the abilities of LLMs to solve formal synthesis benchmarks by carefully crafting a library of prompts for the domain. When one-shot synthesis fails, we propose a novel enumerative synthesis algorithm, which integrates calls to an LLM into a weighted probabilistic search. This allows the synthesizer to provide the LLM with information about the progress of the enumerator, and the LLM to provide the enumerator with syntactic guidance in an iterative loop. We evaluate our techniques on benchmarks from the Syntax-Guided Synthesis (SyGuS) competition. We find that GPT-3.5 as a stand-alone tool for formal synthesis is easily outperformed by state-of-the-art formal synthesis algorithms, but our approach integrating the LLM into an enumerative synthesis algorithm shows significant performance gains over both the LLM and the enumerative synthesizer alone and the winning SyGuS competition tool.

Artificial Intelligence

What problem does this paper attempt to address?

This paper explores how to use large language models (LLMs) to assist in enumerative program synthesis. In formal synthesis tasks, although the current state-of-the-art solutions are based on enumeration algorithms, LLMs have demonstrated excellent performance in generating code from natural language specifications. The researchers evaluate the ability of LLMs to solve benchmark problems in formal synthesis and propose a new method to integrate LLMs into the enumerative synthesis algorithm, providing grammar guidance in the form of probabilistic context-free grammars (pCFG). When a trial fails, this method utilizes the output of LLMs to construct a pCFG and guides the enumerator in an iterative loop. The experiments show that this approach performs better than using LLMs or the enumerator alone, and even surpasses state-of-the-art SyGuS competition tools. The main contributions of this paper include designing a prompt library to guide LLMs in solving formal synthesis problems, proposing a method to guide the enumerator using pCFG generated by LLMs, and introducing a novel CEGIS algorithm that embeds LLMs into the enumerative synthesizer. The results demonstrate that although LLMs have potential in formal program synthesis, they currently work best in combination with existing algorithms.