InstructCoder: Instruction Tuning Large Language Models for Code Editing

Kaixin Li,Qisheng Hu,Xu Zhao,Hui Chen,Yuxi Xie,Tiedong Liu,Qizhe Xie,Junxian He

2024-02-28

Abstract:Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing, containing high-diversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instruction-input-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior code-editing performance matching advanced proprietary LLMs. The datasets and the source code are publicly available at <a class="link-external link-https" href="https://github.com/qishenghu/CodeInstruct" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Software Engineering

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper primarily addresses the following issues: 1. **Limitations of Automatic Code Editing Tools**: Existing automatic code editing tools perform poorly in handling diverse code editing tasks (such as code optimization, refactoring, and fixing), which limits the productivity of developers. This is partly due to the lack of relevant datasets. 2. **Improving the Code Editing Capabilities of Large Language Models (LLMs)**: The authors attempt to enhance the performance of LLMs in real programming tasks by creating a specialized instruction fine-tuning dataset (InstructCoder) for code editing tasks. These tasks include inserting comments, code optimization, and refactoring. 3. **Evaluating the Performance of Existing Models**: The paper introduces a new benchmark dataset (EditEval) to assess the performance of various LLMs on code editing tasks and finds that existing open-source models and some proprietary models perform poorly on these tasks. By introducing the InstructCoder dataset and the EditEval benchmark dataset, the paper demonstrates that instruction fine-tuned open-source LLMs can significantly improve the accuracy of code editing, even rivaling state-of-the-art proprietary models.

InstructCoder: Instruction Tuning Large Language Models for Code Editing

Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions

InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

CodeEditor: Learning to Edit Source Code with Pre-trained Models

InstructEdit: Instruction-based Knowledge Editing for Large Language Models

Instruction Tuning for Large Language Models: A Survey

How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data

CoEdIT: Text Editing by Task-Specific Instruction Tuning

How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with Really Good Data

SelfCodeAlign: Self-Alignment for Code Generation

Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation

DolphCoder: Echo-Locating Code Large Language Models with Diverse and Multi-Objective Instruction Tuning

Dynamics of Instruction Tuning: Each Ability of Large Language Models Has Its Own Growth Pace

Semi-Instruct: Bridging Natural-Instruct and Self-Instruct for Code Large Language Models

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

TeGit: Generating High-Quality Instruction-Tuning Data with Text-Grounded Task Design

BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing

MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

Raw Text is All you Need: Knowledge-intensive Multi-turn Instruction Tuning for Large Language Model

Model Editing for LLMs4Code: How Far are We?