Abstract:The strong performance of large language models (LLMs) on natural language processing tasks raises extensive discussion on their application to code generation. Recent work suggests multiple sampling approaches to improve initial code generation accuracy or program repair approaches to refine the code. However, these methods suffer from LLMs' inefficiencies and limited reasoning capacity. In this work, we propose an LLM programming workflow (LPW) designed to improve both initial code generation and subsequent refinements within a structured two-phase workflow. Specifically, in the solution generation phase, the LLM first outlines a solution plan that decomposes the problem into manageable sub-problems and then verifies the generated solution plan through visible test cases. Subsequently, in the code implementation phase, the LLM initially drafts a code according to the solution plan and its verification. If the generated code fails the visible tests, the plan verification serves as the intended natural language solution to inform the refinement process for correcting bugs. We further introduce SLPW, a sampling variant of LPW, which initially generates multiple solution plans and plan verifications, produces a program for each plan and its verification, and refines each program as necessary until one successfully passes the visible tests. Compared to the state-of-the-art methods across various existing LLMs, our experimental results show that LPW significantly improves the Pass@1 accuracy by up to 16.4% on well-established text-to-code generation benchmarks, especially with a notable improvement of around 10% on challenging benchmarks. Additionally, SLPW demonstrates up to a 5.6% improvement over LPW and sets new state-of-the-art Pass@1 accuracy on various benchmarks, e.g., 98.2% on HumanEval, 84.8% on MBPP, 64.0% on APPS, and 35.3% on CodeContest, using GPT-4o as the backbone.

Programming Languages for Generative Design: A Comparative Study

A visual programming language for drawing and executing flowcharts

General-Purpose Visual Language and Information System with Case-Studies in Developing Business Applications

Text2VP: Generative AI for Visual Programming and Parametric Modeling

Computation for Architecture, hybrid visual and textual language: Research developments and considerations about the implementation of structural imperative and object-oriented paradigms

Visual definition of procedures for automatic virtual scene generation

Automated code development based on genetic programming in graphical programming language: A pilot study

Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages

Survey of Generative Programming

Linguistic Relativity and Programming Languages

Progress and recent trends in generative design

Imperative Genetic Programming

ChatGPT for PLC/DCS Control Logic Generation

A Language for Generic Programming in the Large

PlantoGraphy: Incorporating Iterative Design Process into Generative Artificial Intelligence for Landscape Rendering

Evaluating ChatGPT and GPT-4 for Visual Programming

The PCG: an Empirical Study

Planning-Driven Programming: A Large Language Model Programming Workflow

MyProLang - My Programming Language: A Template-Driven Automatic Natural Programming Language

Enabling Programming Thinking in Large Language Models Toward Code Generation

The importance of visual modelling languages in generative software engineering