Abstract:Financial institutions use credit scorecards for risk management. A scorecard is a data-driven model for predicting default probabilities. Scorecard assessment concentrates on how well a scorecard discriminates good and bad risk. Whether predicted and observed default probabilities agree (i.e., calibration) is an equally important yet often overlooked dimension of scorecard performance. Surprisingly, no attempt has been made to systematically explore different calibration methods and their implications in credit scoring. The goal of the paper is to integrate previous work on probability calibration, to re-introduce available calibration techniques to the credit scoring community, and to empirically examine the extent to which they improve scorecards. More specifically, using real-world credit scoring data, we first develop scorecards using different classifiers, next apply calibration methods to the classifier predictions, and then measure the degree to which they improve calibration. To evaluate performance, we measure the accuracy of predictions in terms of the Brier Score before and after calibration, and employ repeated measures analysis of variance to test for significant differences between group means. Furthermore, we check calibration using reliability plots and decompose the Brier Score to clarify the origin of performance differences across calibrators. The observed results suggest that post-processing scorecard predictions using a calibrator is beneficial. Calibrators improve scorecard calibration while the discriminatory ability remains unaffected. Generalized additive models are particularly suitable for calibrating classifier predictions.

Experiment-based calibration in psychology: Optimal design considerations

The Importance of Calibration in Clinical Psychology

Robust Experimental Designs for Model Calibration

Bayesian Adaptive Calibration and Optimal Design

Designing and evaluating tasks to measure individual differences in experimental psychology: a tutorial

Consensus design of a calibration experiment for human fear conditioning

In pursuit of precision: the calibration of minds and machines in late nineteenth-century psychology

Optimal design of experiments to identify latent behavioral types

Calibrating Model-Based Inferences and Decisions

Primacy of theory? Exploring perspectives on validity in conceptual psychometrics

Approaches for credit scorecard calibration: An empirical analysis

A Unifying Theory of Distance from Calibration

Optimal Design of Validation Experiments for the Prediction of Quantities of Interest

Towards reliable predictive analytics: a generalized calibration framework

A Large-Scale Study of Probabilistic Calibration in Neural Network Regression

Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors

Calibration Scoring Rules for Practical Prediction Training

Metrics for Bayesian Optimal Experiment Design under Model Misspecification

Beyond Static Calibration: The Impact of User Preference Dynamics on Calibrated Recommendation

The Road to Hell Is Paved With Good Intentions: How Common Practices in Scale Construction Hurt Validity

Reducing Attenuation Bias in Regression Analyses Involving Rating Scale Data via Psychometric Modeling