Abstract:Code review is an important practice in software development. One of its main objectives is for the assurance of code quality. For this purpose, the efficacy of code review is subject to the credibility of reviewers, i.e, reviewers who have demonstrated strong evidence of previously making quality-enhancing comments are more credible than those who have not. Code reviewer recommendation (CRR) is designed to assist in recommending suitable reviewers for a specific objective and, in this context, assurance of code quality. Its performance is susceptible to the relevance of its training dataset to this objective, composed of all reviewers’ historical review comments, which, however, often contains a plethora of comments that are irrelevant to the enhancement of code quality. Furthermore, recommendation accuracy has been adopted as the sole metric to evaluate a recommender’s performance, which is inadequate as it does not take reviewers’ relevant credibility into consideration. These two issues form the ground truth problem in CRR as they both originate from the relevance of dataset used to train and evaluate CRR algorithms. To tackle this problem, we first propose the concept of Quality-Enhancing Review Comments (QERC), which includes three types of comments - change-triggering inline comments, informative general comments, and approve-to-merge comments. We then devise a set of algorithms and procedures to obtain a distilled dataset by applying QERC to the original dataset. We finally introduce a new metric – reviewer’s credibility for quality enhancement (RCQE) – as a complementary metric to recommendation accuracy for evaluating the performance of recommenders. To validate the proposed QERC-based approach to CRR, we conduct empirical studies using real data from seven projects containing over 82K pull requests and 346K review comments. Results show that: (a) QERC can effectively address the ground truth problem by distilling quality-enhancing comments from the dataset containing original code reviews, (b) QERC can assist recommenders in finding highly credible reviewers at a slight cost of recommendation accuracy, and (c) even “wrong” recommendations using the distilled dataset are likely to be more credible than those using the original dataset.

GPP: A Graph-Powered Prioritizer for Code Review Requests

Prioritizing code review requests to improve review efficiency: a simulation study

Early Prediction of Merged Code Changes to Prioritize Reviewing Tasks

Recommending Code Reviewers for Proprietary Software Projects: A Large Scale Study

Code Reviewer Recommendation in Tencent

Evaluating Learning-to-Rank Models for Prioritizing Code Review Requests Using Process Simulation

Efficient priority rules for the stochastic resource-constrained project scheduling problem

Optimizing test prioritization via test distribution analysis.

An Iterative Algorithm to Derive Priority from Large-Scale Sparse Pairwise Comparison Matrix

A novel priority dispatch rule generation method based on graph neural network and reinforcement learning for distributed job-shop scheduling

Distilling Quality Enhancing Comments from Code Reviews to Underpin Reviewer Recommendation

An Empirical Study Towards Prompt-Tuning for Graph Contrastive Pre-Training in Recommendations.

Redefining Crowdsourced Test Report Prioritization: An Innovative Approach with Large Language Model

Graph-Enhanced Prompt Learning for Personalized Review Generation

Code Reviewer Recommendation Based on a Hypergraph with Multiplex Relationships

Modeling Review History for Reviewer Recommendation: A Hypergraph Approach

Prioritizing documentation effort: Can we do better?

A multiobjective stochastic genetic algorithm for the pareto-optimal prioritization scheme design of real-time healthcare resource allocation

PAID: Prioritizing app issues for developers by tracking user reviews over versions

A Survey of Graph Pre-processing Methods: From Algorithmic to Hardware Perspectives

Prioritizing Code Documentation Effort: Can We Do It Simpler but Better?