Abstract:Just-In-Time Software Defect Prediction (JIT-SDP) uses machine learning to predict whether software changes are defect-inducing or clean. When adopting JIT-SDP, changes in the underlying defect generating process may significantly affect the predictive performance of JIT-SDP models over time. Therefore, being able to continuously track the predictive performance of JIT-SDP models during the software development process is of utmost importance for software companies to decide whether or not to trust the predictions provided by such models over time. However, there has been little discussion on how to continuously evaluate predictive performance in practice, and such evaluation is not straightforward. In particular, labeled software changes that can be used for evaluation arrive over time with a delay, which in part corresponds to the time we have to wait to label software changes as ‘clean’ (waiting time). A clean label assigned based on a given waiting time may not correspond to the true label of the software changes. This can potentially hinder the validity of any continuous predictive performance evaluation procedure for JIT-SDP models. This paper provides the first discussion of how to continuously evaluate predictive performance of JIT-SDP models over time during the software development process, and the first investigation of whether and to what extent waiting time affects the validity of such continuous performance evaluation procedure in JIT-SDP. Based on 13 GitHub projects, we found that waiting time had a significant impact on the validity. Though typically small, the differences in estimated predicted performance were sometimes large, and thus inappropriate choices of waiting time can lead to misleading estimations of predictive performance over time. Such impact did not normally change the ranking between JIT-SDP models, and thus conclusions in terms of which JIT-SDP model performs better are likely reliable independent of the choice of waiting time, especially when considered across projects.

Towards Just-In-Time Feature Request Approval Prediction

Early Prediction of Merged Code Changes to Prioritize Reviewing Tasks

Utilizing Creator Profiles for Predicting Valuable User Enhancement Reports

The Need for a Fine-grained approach in Just-in-Time Defect Prediction

Effort-aware and just-in-time defect prediction with neural network

Human-in-the-loop online just-in-time software defect prediction

A Systematic Survey of Just-in-Time Software Defect Prediction

Deep Learning for Just-In-Time Defect Prediction

A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development

Just-in-time Defect Prediction for Software Hunks

An Empirical Study on the Impact of Class Overlapin Just-in-Time Software Defect Prediction (S).

On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction

DEJIT: A Differential Evolution Algorithm for Effort-Aware Just-in-Time Software Defect Prediction

Automatically Classifying User Requests in Crowdsourcing Requirements Engineering

Feature Sets in Just-in-Time Defect Prediction: An Empirical Evaluation

Intelligent Request Strategy Design in Recommender System

IRJIT: A Simple, Online, Information Retrieval Approach for Just-In-Time Software Defect Prediction

Effort-Aware semi-Supervised just-in-Time defect prediction

Towards Reliable Online Just-in-Time Software Defect Prediction

A Deep Classifier for Crowdsourcing User Requests

Just‐in‐time Defect Prediction Enhanced by the Joint Method of Line Label Fusion and File Filtering