Abstract:Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at <a class="link-external link-https" href="https://github.com/evalplus/evalplus" rel="external noopener nofollow">this https URL</a> to facilitate and accelerate future LLM-for-code research.

Synthesizing Contracts Correct Modulo a Test Generator.

Learning Stateful Preconditions Modulo a Test Generator

Synthesizing LTL contracts from component libraries using rich counterexamples

Smart Contract Synthesis Modulo Hyperproperties

Contract-Based Distributed Synthesis in Two-Objective Parity Games

Isyn: Semi-automated Smart Contract Synthesis from Legal Financial Agreements

Online Test Synthesis From Requirements: Enhancing Reinforcement Learning with Game Theory

Differential testing solidity compiler through deep contract manipulation and mutation

Towards Understanding Contracts Grammar: A Large Language Model-Based Extractive Question-Answering Approach

Schooling to Exploit Foolish Contracts

Certified Compilation of Financial Contracts

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

Securing Smart Contract On The Fly

Synthesizing Formal Semantics from Executable Interpreters

Towards Automated Verification of LLM-Synthesized C Programs

Program synthesis using conflict-driven learning

Detecting Buggy Contracts via Smart Testing

Synthesis from LTL with Reward Optimization in Sampled Oblivious Environments

Effective Targeted Testing of Smart Contracts

Synthesizing Method Sequences for High-Coverage Testing