Abstract:The strong performance of large language models (LLMs) on natural language processing tasks raises extensive discussion on their application to code generation. Recent work suggests multiple sampling approaches to improve initial code generation accuracy or program repair approaches to refine the code. However, these methods suffer from LLMs' inefficiencies and limited reasoning capacity. In this work, we propose an LLM programming workflow (LPW) designed to improve both initial code generation and subsequent refinements within a structured two-phase workflow. Specifically, in the solution generation phase, the LLM first outlines a solution plan that decomposes the problem into manageable sub-problems and then verifies the generated solution plan through visible test cases. Subsequently, in the code implementation phase, the LLM initially drafts a code according to the solution plan and its verification. If the generated code fails the visible tests, the plan verification serves as the intended natural language solution to inform the refinement process for correcting bugs. We further introduce SLPW, a sampling variant of LPW, which initially generates multiple solution plans and plan verifications, produces a program for each plan and its verification, and refines each program as necessary until one successfully passes the visible tests. Compared to the state-of-the-art methods across various existing LLMs, our experimental results show that LPW significantly improves the Pass@1 accuracy by up to 16.4% on well-established text-to-code generation benchmarks, especially with a notable improvement of around 10% on challenging benchmarks. Additionally, SLPW demonstrates up to a 5.6% improvement over LPW and sets new state-of-the-art Pass@1 accuracy on various benchmarks, e.g., 98.2% on HumanEval, 84.8% on MBPP, 64.0% on APPS, and 35.3% on CodeContest, using GPT-4o as the backbone.

An Opportunistically Parallel Lambda Calculus for Performant Composition of Large Language Models

HPC-Coder: Modeling Parallel Programs using Large Language Models

Can Large Language Models Write Parallel Code?

An LLM-Tool Compiler for Fused Parallel Function Calling

Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

OccamLLM: Fast and Exact Language Model Arithmetic in a Single Step

Perplexed: Understanding When Large Language Models are Confused

LLM Augmented LLMs: Expanding Capabilities through Composition

Interpreting and Improving Large Language Models in Arithmetic Calculation

Large Language Models as Code Executors: An Exploratory Study

CLLMs: Consistency Large Language Models

Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference

Meta Large Language Model Compiler: Foundation Models of Compiler Optimization

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

From Words to Worlds: Compositionality for Cognitive Architectures

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Large Language Models for Compiler Optimization

Tandem Transformers for Inference Efficient LLMs

Code Simulation Challenges for Large Language Models

Quantum Many-Body Physics Calculations with Large Language Models

Planning-Driven Programming: A Large Language Model Programming Workflow