LEVER: Learning to Verify Language-to-Code Generation with Execution

Ansong Ni,Srini Iyer,Dragomir Radev,Ves Stoyanov,Wen-tau Yih,Sida I. Wang,Xi Victoria Lin

DOI: https://doi.org/10.48550/arXiv.2302.08468

IF: 5.414

2023-02-16

Machine Learning

Abstract:The advent of pre-trained code language models (CodeLMs) has lead to significant progress in language-to-code generation. State-of-the-art approaches in this area combine CodeLM decoding with sample pruning and reranking using test cases or heuristics based on the execution results. However, it is challenging to obtain test cases for many real-world language-to-code applications, and heuristics cannot well capture the semantic features of the execution results, such as data type and value range, which often indicates the correctness of the program. In this work, we propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the CodeLM is correct or not based on the natural language input, the program itself and its execution results. The sampled programs are reranked by combining the verification score with the CodeLM generation probability, and marginalizing over programs with the same execution results. On four datasets across the domains of table QA, math QA and basic Python programming, LEVER consistently improves over the base CodeLMs (4.6% to 10.9% with code-davinci-002) and achieves new state-of-the-art results on all of them.

What problem does this paper attempt to address?

The paper aims to address the accuracy issue in the task of natural language to code generation. Specifically, although existing large language models (LLMs) have shown significant progress in generating code, there are still challenges in practical applications, especially in the absence of test cases and the difficulty in accurately capturing program semantic features through execution results. The paper proposes the LEVER method, a simple approach that improves the quality of natural language to code generation by learning to verify the generated programs and their execution results. LEVER trains a verifier to judge whether the programs sampled from LLMs are correct and re-ranks the candidate programs by combining generation probability and verification scores.

This method has been experimentally validated on datasets from four different domains, including table QA, mathematical reasoning, and basic Python programming, all of which have shown significant performance improvements. Notably, when combined with code-davinci-002, LEVER outperformed the existing best models on all benchmarks and demonstrated good performance in resource-limited or weakly supervised environments. Additionally, the paper explores the performance of LEVER with different training sample sizes and demonstrates its effectiveness with a small amount of data.