Improving Automated Code Reviews: Learning from Experience

Hong Yi Lin,Patanamon Thongtanunam,Christoph Treude,Wachiraphan Charoenwet

DOI: https://doi.org/10.1145/3643991.3644910

2024-02-06

Abstract:Modern code review is a critical quality assurance process that is widely adopted in both industry and open source software environments. This process can help newcomers learn from the feedback of experienced reviewers; however, it often brings a large workload and stress to reviewers. To alleviate this burden, the field of automated code reviews aims to automate the process, teaching large language models to provide reviews on submitted code, just as a human would. A recent approach pre-trained and fine-tuned the code intelligent language model on a large-scale code review corpus. However, such techniques did not fully utilise quality reviews amongst the training data. Indeed, reviewers with a higher level of experience or familiarity with the code will likely provide deeper insights than the others. In this study, we set out to investigate whether higher-quality reviews can be generated from automated code review models that are trained based on an experience-aware oversampling technique. Through our quantitative and qualitative evaluation, we find that experience-aware oversampling can increase the correctness, level of information, and meaningfulness of reviews generated by the current state-of-the-art model without introducing new data. The results suggest that a vast amount of high-quality reviews are underutilised with current training strategies. This work sheds light on resource-efficient ways to boost automated code review models.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the quality of automated code review. Specifically, the authors focus on how to train an automated code review model through an experience - aware oversampling technique to generate higher - quality code review comments. Existing methods do not fully utilize high - quality review comments when training models, especially those from experienced reviewers. The authors assume that by increasing the training weights of the comments provided by these experienced reviewers, the correctness, informativeness, and meaningfulness of the comments generated by the model can be improved without introducing new data.

To verify this hypothesis, the authors designed a series of experiments, using the CodeReviewer model and fine - tuning it through different oversampling strategies (such as oversampling only for the main author of the main reviewer, all main reviewers, or all main authors). The experimental results show that the experience - aware oversampling model can generate more semantically correct comments, provide more suggestions and explanations, and can comment on more critical issues, especially issues in logic, validation, and resource management. This indicates that by oversampling the comments of experienced reviewers, the performance of the automated code review model can be significantly improved without adding new data.