Abstract:The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks. Our dataset APPS+ and StepCoder are available online.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced by large - language models (LLMs) in optimizing the quality of code generation from compiler feedback through reinforcement learning (RL) when faced with complex human requirements in code - generation tasks. Specifically, the paper focuses on two main issues: 1. **Exploration Difficulties**: Due to the complexity of human requirements, the code sequences generated by LLMs are often long, which makes RL exploration difficult. Long code sequences increase the difficulty of exploration because the correct code path needs to be found to meet complex programming requirements. 2. **Optimization Difficulties**: In some cases, a single unit test may not cover complex code, resulting in the emergence of unexecuted code fragments. These unexecuted code fragments are not related to the final reward, so optimizing based on the entire code sequence may be inaccurate. In addition, existing datasets for RL training (such as APPS) have quality problems, which affect the accuracy of learning from compiler feedback through RL. To solve these problems, the paper proposes the StepCoder framework, which contains two main components: - **Curriculum of CodeCompletion Subtasks (CCCS)**: By decomposing complex code - generation tasks into a series of simpler code - completion subtasks and gradually increasing the difficulty of the tasks, the exploration process is simplified. - **Fine - Grained Optimization (FGO)**: Through dynamic masking technology, only the code fragments executed in the unit test are optimized, avoiding the influence of unexecuted code fragments on optimization and improving the optimization precision. In addition, the paper also constructs a high - quality dataset APPS + for RL training to ensure the correctness of unit tests and the quality of the dataset. Experimental results show that StepCoder is superior to other methods in terms of exploration efficiency and effectiveness and performs well in multiple benchmarks.

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback

Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

JumpCoder: Go Beyond Autoregressive Coder via Online Modification

RLCoder: Reinforcement Learning for Repository-Level Code Completion

Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement

ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generation

Towards Enhancing In-Context Learning for Code Generation

CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules

RLTF: Reinforcement Learning from Unit Test Feedback

Compilable Neural Code Generation with Compiler Feedback

Execution-based Code Generation using Deep Reinforcement Learning

Effi-Code: Unleashing Code Efficiency in Language Models

A Pair Programming Framework for Code Generation Via Multi-Plan Exploration and Feedback-Driven Refinement

SelfEvolve: A Code Evolution Framework via Large Language Models

FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system

Think Outside the Code: Brainstorming Boosts Large Language Models in Code Generation

Code Generation Using Self-Interactive Assistant