CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis

Erik Nijkamp,Bo Pang,Hiroaki Hayashi,Lifu Tu,Huan Wang,Yingbo Zhou,Silvio Savarese,Caiming Xiong

2023-02-28

Abstract:Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: <a class="link-external link-https" href="https://github.com/salesforce/CodeGen" rel="external noopener nofollow">this https URL</a>.

Machine Learning,Computation and Language,Programming Languages

What problem does this paper attempt to address?

The paper mainly addresses two core issues:

1. **Enhancing Program Synthesis Capability**: By developing large-scale language models (referred to as CODEGEN), it aims to improve the ability to automatically generate computer programs from natural language descriptions. These models are trained to understand and generate code, particularly excelling in the Python programming language.

2. **Multi-round Program Synthesis Method**: A new multi-round program synthesis method is proposed, allowing users to provide requirement specifications to the system step-by-step in natural language, while the system generates corresponding subprograms in steps. This method simplifies the complexity of user input and increases the likelihood of the model generating correct programs.

To achieve the above goals, the authors conducted the following work:

- Developed a series of language models of different scales (350M to 16.1B parameters) and trained them on various datasets, including natural language data and programming language data.
- Proposed a multi-round program synthesis benchmark (MTPB) to evaluate the performance of models in multi-step program synthesis tasks.
- Analysis shows that the multi-round program synthesis method significantly improves the success rate of program synthesis, especially in more complex tasks.

In summary, the main contribution of this paper lies in proposing large-scale language models that better understand user intent and generate programs through multi-round interactions, as well as a new benchmark for evaluating this capability.