Abstract:Software engineers mainly write code by editing existing programs. In contrast, large language models (LLMs) autoregressively synthesize programs in a single pass. One explanation for this is the scarcity of open-sourced edit data. While high-quality instruction data for code synthesis is already scarce, high-quality edit data is even scarcer. To fill this gap, we develop a synthetic data generation algorithm called LintSeq. This algorithm refactors existing code into a sequence of code edits by using a linter to procedurally sample across the error-free insertions that can be used to sequentially write programs. It outputs edit sequences as text strings consisting of consecutive program diffs. To test LintSeq, we use it to refactor a dataset of instruction + program pairs into instruction + program-diff-sequence tuples. Then, we instruction finetune a series of smaller LLMs ranging from 2.6B to 14B parameters on both the re-factored and original versions of this dataset, comparing zero-shot performance on code synthesis benchmarks. We show that during repeated sampling, edit sequence finetuned models produce more diverse programs than baselines. This results in better inference-time scaling for benchmark coverage as a function of samples, i.e. the fraction of problems "pass@k" solved by any attempt given "k" tries. For example, on HumanEval pass@50, small LLMs finetuned on synthetic edit sequences are competitive with GPT-4 and outperform models finetuned on the baseline dataset by +20% (+/-3%) in absolute score. Finally, we also pretrain our own tiny LMs for code understanding. We show that finetuning tiny models on synthetic code edits results in state-of-the-art code synthesis for the on-device model class. Our 150M parameter edit sequence LM matches or outperforms code models with twice as many parameters, both with and without repeated sampling, including Codex and AlphaCode.

Learning Program Synthesis for Integer Sequences from Scratch

Alien coding

Stepping Stones to Inductive Synthesis of Low-Level Looping Programs

FACT: Learning Governing Abstractions Behind Integer Sequences

Benchmarking Large Language Models with Integer Sequence Generation Tasks

Computing Integer Sequences: Filtering vs Generation (Functional Pearl)

Another Set of Sequences, Sub-Sequences, and Sequences of Sequences

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

How to Create a New Integer Sequence

Latent Programmer: Discrete Latent Codes for Program Synthesis

A Scheme-Driven Approach to Learning Programs from Input/Output Equations

Discovery of Exact Equations for Integer Sequences

Iterative Genetic Improvement: Scaling Stochastic Program Synthesis

Improved Tree Search for Automatic Program Synthesis

Synthesizing Method Sequences for High-Coverage Testing

Integer Sequences and Output Arrays

A Mathematical Benchmark for Inductive Theorem Provers

Neural Program Search: Solving Programming Tasks from Description and Examples

Program Synthesis using Natural Language

Inductive Program Synthesis via Iterative Forward-Backward Abstract Interpretation

Fe b 20 18 Learning to Synthesize