Abstract:Abstract Objective After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. Materials and Methods We introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive “Type I and II” regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss. Results Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818–0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230–0.300) and 0.241 (95%CI, 0.216–0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765–0.769) to 0.800 (95%CI, 0.798–0.802) and 0.799 (95%CI, 0.797–0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913–0.935) and 0.925 (95%CI, 0.914–0.935), compared to the static model’s aAUC of 0.904 (95%CI, 0.892–0.916). Discussion Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data. Conclusions BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time.

BRcal: An R Package to Boldness-Recalibrate Probability Predictions

pencal: an R Package for the Dynamic Prediction of Survival with Many Longitudinal Predictors

Cautious Calibration in Binary Classification

Calibrate: Interactive Analysis of Probabilistic Model Output

CaliCo: a R package for Bayesian calibration

abc: an R package for Approximate Bayesian Computation (ABC)

Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

Beyond Binary Preference: Leveraging Bayesian Approaches for Joint Optimization of Ranking and Calibration

Probabilistic Scores of Classifiers, Calibration is not Enough

Calibrar: an R package for fitting complex ecological models

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Risk-based Calibration for Probabilistic Classifiers

Probabilistic recalibration of forecasts

unmconf : an R package for Bayesian regression with unmeasured confounders

AdaptiveConformal: An R Package for Adaptive Conformal Inference

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

BayesVarSel: Bayesian Testing, Variable Selection and model averaging in Linear Models using R

Online Calibrated and Conformal Prediction Improves Bayesian Optimization

Towards reliable predictive analytics: a generalized calibration framework

Bayesian Safety Validation for Failure Probability Estimation of Black-Box Systems