Abstract:Developer turnover is inevitable on software projects and leads to knowledge loss, a reduction in productivity, and an increase in defects. Mitigation strategies to deal with turnover tend to disrupt and increase workloads for developers. In this work, we suggest that through code review recommendation we can distribute knowledge and mitigate turnover while more evenly distributing review workload. We conduct historical analyses to understand the natural concentration of review workload and the degree of knowledge spreading that is inherent in code review. Even though review workload is highly concentrated, we show that code review natural spreads knowledge thereby reducing the files at risk to turnover. Using simulation, we evaluate existing code review recommenders and develop novel recommenders to understand their impact on the level of expertise during review, the workload of reviewers, and the files at risk to turnover. Our simulations use seeded random replacement of reviewers to allow us to compare the reviewer recommenders without the confounding variation of different reviewers being replaced for each recommender. We find that prior work that assigns reviewers based on file ownership concentrates knowledge on a small group of core developers increasing the risk of knowledge loss from turnover. Recent work, WhoDo, that considers developer workload, assigns developers that are not sufficiently committed to the project and we see an increase in files at risk to turnover. We propose learning and retention aware review recommenders that when combined are effective at reducing the risk of turnover, but they unacceptably reduce the overall expertise during reviews. Combining recommenders, we develop the SofiaWL recommender that suggests experts with low active review workload when none of the files under review are known by only one developer. In contrast, when knowledge is concentrated on one developer, it sends the review to other reviewers to spread knowledge. For the projects we study, we are able to globally increase expertise during reviews, $+3$+3%, reduce workload concentration, $-12$−12%, and reduce the files at risk, $-28$−28%. We make our scripts and data available in our replication package [1]. Developers can optimize for a particular outcome measure based on the needs of their project, or use our GitHub bot to automatically balance the outcomes [2].

Towards debiasing code review support

Debiasing Judgements Using a Distributed Cognition Approach: A Scoping Review of Technological Strategies

Understanding and effectively mitigating code review anxiety

Rolling in the deep of cognitive and AI biases

Improving Code Reviewer Recommendation: Accuracy, Latency, Workload, and Bystanders

Bias and Debias in Recommender System: A Survey and Future Directions

An Industrial Case Study on Shrinking Code Review Changesets through Remark Prediction

Advancing Modern Code Review Effectiveness through Human Error Mechanisms

The Importance of Cognitive Biases in the Recommendation Ecosystem

D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias

'Propose and Review': Interactive Bias Mitigation for Machine Classifiers

Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?

Studying the impact of risk assessment analytics on risk awareness and code review performance

Code Reviews in Open Source Projects : How Do Gender Biases Affect Participation and Outcomes?

A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models

Uncovering and Quantifying Social Biases in Code Generation

Using StackOverflow content to assist in code review

Revisiting Technical Bias Mitigation Strategies

Debiasing Evaluations That are Biased by Evaluations

Factoring Expertise, Workload, and Turnover into Code Review Recommendation

Bias-Aware Design for Informed Decisions: Raising Awareness of Self-Selection Bias in User Ratings and Reviews