Abstract:Software engineers mainly write code by editing existing programs. In contrast, large language models (LLMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of open-sourced edit data. While high-quality instruction data for code synthesis is already scarce, high-quality edit data is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors existing code into a sequence of code edits by using a linter to procedurally sample across the error-free insertions that can be used to sequentially write programs. It outputs edit sequences as text strings consisting of consecutive program diffs. To test LintSeq, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we instruction finetune a series of smaller LLMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset, comparing zero-shot performance on code synthesis benchmarks. We show that during repeated sampling, edit sequence finetuned models produce more diverse programs than baselines. This results in better inference-time scaling for benchmark coverage as a function of samples, i.e. the fraction of problems "pass@k" solved by any attempt given "k" tries. For example, on HumanEval pass@50, small LLMs finetuned on synthetic edit sequences are competitive with GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%) in absolute score. Finally, we also pretrain our own tiny LMs for code understanding. We show that finetuning tiny models on synthetic code edits results in state-of-the-art code synthesis for the on-device model class. Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters, both with and without repeated sampling, including Codex and AlphaCode.

Alien coding

Learning Program Synthesis for Integer Sequences from Scratch

Open-Ended Automatic Programming Through Combinatorial Evolution

NoviCode: Generating Programs from Natural Language Utterances by Novices

code2seq: Generating Sequences from Structured Representations of Code

Latent Programmer: Discrete Latent Codes for Program Synthesis

AI Programmer: Autonomously Creating Software Programs Using Genetic Algorithms

Stepping Stones to Inductive Synthesis of Low-Level Looping Programs

EgoCoder: Intelligent Program Synthesis with Hierarchical Sequential Neural Network Model

Magicoder: Source Code Is All You Need

Self-Programming Artificial Intelligence Using Code-Generating Language Models

Automatic Program Synthesis of Long Programs with a Learned Garbage Collector

Neural Program Search: Solving Programming Tasks from Description and Examples

Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code

Gene Expression Programming: a New Adaptive Algorithm for Solving Problems

Learning to Reason via Program Generation, Emulation, and Search

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment

Competition-level code generation with AlphaCode

OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code