Abstract:Large language models (LLMs) have already revolutionized code generation, after being pretrained on publicly available code data. However, while various methods have been proposed to augment LLMs with retrieved knowledge and enhance the quality of code generation, the performance of these retrieval-based methods is limited by the strength of the retrievers used. In addition, while LLMs show great emergent ability, they still struggle to produce the correct code in one turn. To address these challenges, we propose a novel two-step pipeline, called \autoknow, that leverages LLMs as both knowledge providers and self-reflective programmers. Unlike retrieval-based methods, \autoknow~obtains the knowledge from input prompts and generates intermediate code based on the generated knowledge. After that, \autoknow~asks LLM to act as an expert programmer to perform debugging for the generated code. This is achieved by receiving the error message from the interpreter, without requiring special test cases for correctness verification. We evaluate \autoknow~on three code generation datasets, including DS-1000 for data science code, HumanEval for software engineering code, and TransCoder for C++-to-Python translation. Our empirical experiments show that \autoknow~outperforms strong baselines by a significant margin on all datasets. We also conduct exhaustive analytical experiments to validate the effectiveness of the two stages of \autoknow, and find that both are superior to other prompting-based methods. Further scalability analysis demonstrates that \autoknow~can be adapted to other more advanced models, such as GPT-4, and bring consistent efficacy improvement.

Embedded-DSL-Like Code Generation and Optimization of Bayesian Estimation Routines with User-Defined Source-to-Source Code Transformation Framework Xevolver

Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation

Automated Deep Learning Optimization via DSL-Based Source Code Transformation

DOCE: Finding the Sweet Spot for Execution-Based Code Generation

Composing inference algorithms as program transformations

SelfEvolve: A Code Evolution Framework via Large Language Models

JumpCoder: Go Beyond Autoregressive Coder via Online Modification

Co-evolutionary automatic programming for software development

Effi-Code: Unleashing Code Efficiency in Language Models

Linguacodus: A Synergistic Framework for Transformative Code Generation in Machine Learning Pipelines

Doubly Bayesian Optimization

DynEx: Dynamic Code Synthesis with Structured Design Exploration for Accelerated Exploratory Programming

Design of an efficient Transformer-XL model for enhanced pseudo code to Python code conversion

A Pair Programming Framework for Code Generation Via Multi-Plan Exploration and Feedback-Driven Refinement

A Generator Framework For Evolving Variant-Rich Software

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

StructCoder: Structure-Aware Transformer for Code Generation

Bias and Error Mitigation in Software-Generated Data: An Advanced Search and Optimization Framework Leveraging Generative Code Models

Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought

evosax: JAX-based Evolution Strategies