Exploring the Potential of Pre-Trained Language Models of Code for Automated Program Repair

Sichong Hao,Xianjun Shi,Hongwei Liu

DOI: https://doi.org/10.3390/electronics13071200

IF: 2.9

2024-03-26

Electronics

Abstract:In the realm of software development, automated program repair (APR) emerges as a pivotal technique, autonomously debugging faulty code to boost productivity. Despite the notable advancements of large pre-trained language models of code (PLMCs) in code generation, their efficacy in complex tasks like APR remains suboptimal. This limitation is attributed to the generic development of PLMCs, whose specialized potential for APR is yet be to fully explored. In this paper, we propose a novel approach designed to enhance PLMCs' APR performance through source code augmentation and curriculum learning. Our approach employs code augmentation operators to generate a spectrum of syntactically varied yet semantically congruent bug-fixing programs, thus enriching the dataset's diversity. Furthermore, we design a curriculum learning strategy, enabling PLMCs to develop a deep understanding of program semantics from these enriched code variants, thereby refining their APR fine-tuning prowess. We apply our approach across different PLMCs and systematically evaluate it on three benchmarks: BFP-small, BFP-medium, and Defects4J. The experimental results show that our approach outperforms both original models and existing baseline methods, demonstrating the promising future of adapting PLMCs for code debugging in practice.

engineering, electrical & electronic,computer science, information systems,physics, applied

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the performance of large pre - trained language models of code (PLMCs) in the automated program repair (APR) task. Although PLMCs have made remarkable progress in code generation, their effectiveness is still not satisfactory when dealing with complex tasks such as automated program repair. The paper points out that this limitation is mainly due to the general - purpose development of PLMCs, and their potential specifically for APR has not been fully explored.

To meet this challenge, the paper proposes a new method to enhance the performance of PLMCs in APR tasks through source - code augmentation and curriculum learning. Specifically:

1. **Source - code augmentation**: Use code - augmentation operators to generate a series of error - fixing programs with diverse syntax but consistent semantics, thereby enriching the diversity of the data set.
2. **Curriculum learning**: Design a curriculum - learning strategy to enable PLMCs to deeply understand program semantics from these augmented code variants, thereby optimizing their APR fine - tuning ability.

Through this method, the paper aims to increase the success rate of PLMCs in automated program - repair tasks, especially the code - debugging ability in practical applications. Experimental results show that this method outperforms the original model and other existing baseline methods on multiple benchmarks.