Abstract:This work introduces self-infilling code generation, a general framework that incorporates infilling operations into auto-regressive decoding. Our approach capitalizes on the observation that recent infilling-capable code language models can self-infill: whereas infilling operations aim to fill in the middle based on a predefined prefix and suffix, self-infilling sequentially generates both such surrounding context and the infilled content. We utilize this capability to introduce novel interruption and looping mechanisms in conventional decoding, evolving it into a non-monotonic process. Interruptions allow for postponing the generation of specific code until a definitive suffix is established, enhancing control over the output. Meanwhile, the looping mechanism, which leverages the complementary nature of self-infilling and left-to-right decoding, can iteratively update and synchronize each piece of generation cyclically. Extensive experiments are conducted to demonstrate that our proposed decoding process is effective in enhancing both regularity and quality across several code generation benchmarks.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to improve the decoding process of code - generation models by introducing the self - infilling mechanism, thereby enhancing the quality and consistency of the generated code.
Specifically, most existing code - generation models adopt a strict left - to - right decoding method, which limits their performance when dealing with tasks that require bidirectional context. For example, in tasks such as partial code completion, docstring generation, and type prediction, traditional decoding methods may lead to inaccurate or inconsistent generation results due to the inability to fully utilize the preceding and following information. In addition, traditional methods are prone to the problem of error propagation (exposure bias) in high - entropy situations, that is, the deviation of the subsequently generated content due to the uncertainty of predicting the next token.
To solve these problems, this paper proposes a new framework - self - infilling code generation. The core idea of this framework is to incorporate self - infilling operations into the auto - regressive decoding process, enabling the model to dynamically generate surrounding context and filling content. In this way, the author introduces an interruption mechanism and a looping mechanism:
1. **Interruption Mechanism**:
- When the model encounters an uncertain situation during the decoding process, it can temporarily interrupt the generation, first generate a definite suffix, and then return to the interruption point for filling.
- This mechanism helps to alleviate the exposure bias problem and avoid generation bias caused by incorrect context.
2. **Looping Mechanism**:
- By alternately using self - infilling and left - to - right conditional generation, the model can update fragments in each iteration and gradually synchronize the intermediate part with the latest suffix information.
- This mechanism enables each fragment to be repeatedly updated in a richer context, thereby improving the overall quality and consistency of generation.
The experimental results show that self - infilling code generation not only significantly improves the quality of the generated code but also effectively reduces the degeneration phenomenon, that is, the generation of empty code or duplicate code. In addition, this method performs well in multiple code - generation benchmark tests, especially demonstrating its flexibility and adaptability in multi - language code - generation tasks.
In summary, this paper aims to improve the decoding process of existing code - generation models by introducing the self - infilling mechanism, making them more flexible and better able to utilize bidirectional context information, thereby generating code with higher quality and consistency.