Abstract:Subject. This article deals with the issues of validation of the consistency of rating-based model forecasts. Objectives. The article aims to provide developers and validators of rating-based models with a practical fundamental test for benchmarking study of the estimated default probability values obtained as a result of the application of models used in the rating system. Methods. For the study, I used the classical interval approach to testing of statistical hypotheses focused on the subject area of calibration of rating systems. Results. In addition to the generally accepted tests for the correspondence of the predicted probabilities of default of credit risk objects to the historically realized values, the article proposes a new statistical test that corrects the shortcomings of the generally accepted ones, focused on "diagnosing" the consistency of the implemented discrimination of objects by the rating model. Examples of recognizing the reasons for a negative test result and negative consequences for lending are given while maintaining the current settings of the rating model. In addition to the bias in the assessment of the total frequency of defaults in the loan portfolio, the proposed method makes it possible to objectively reveal the inadequacy of discrimination against borrowers with a calibrated rating model, diagnose the “disease” of the rating model. Conclusions and Relevance. The new practical benchmark test allows to reject the hypothesis about the consistency of assessing the probability of default by the rating model at a given level of confidence and available historical data. The test has the advantage of practical interpretability based on its results, it is possible to draw a conclusion about the direction of the model correction. The offered test can be used in the process of internal validation by the bank of its own rating models, which is required by the Bank of Russia for approaches based on internal ratings.

A hypothesis test for the long-term calibration in rating systems with overlapping time windows

Benchmark test for calculated default probability values derived from the application of rating-based models

Uniform reliability tests for forecasting systems with small lead time

Calibration of the rating transition model for high and low default portfolios

Diagnostic Checking for the Adequacy of Nonlinear Time Series Models

Testing for Structural Breaks – A New Self-Normalization Approach Based on the Adjusted Sample Range

A Safe Hosmer-Lemeshow Test

Approach to Testing Nonlinear Time Series Models

A New Method of Measuring Default Probability with Complete Information in Credit Ratings

Calibration of Machine Learning Classifiers for Probability of Default Modelling

Testing Calibration in Nearly-Linear Time

A critical review of existing and new population stability testing procedures in credit risk scoring

Nonparametric Specification Testing for Continuous-Time Models with Application to Spot Interest Rates

Characteristic Function-Based Testing for Multifactor Continuous-Time Markov Models Via Nonparametric Regression

Phase transition in the Bayesian estimation of the default portfolio

Approaches for credit scorecard calibration: An empirical analysis

A Proposed Simulation Technique for Population Stability Testing in Credit Risk Scorecards

Testing For Auto-Calibration With Lorenz And Concentration Curves

Remarks on the monotonicity of default probabilities

Assessing the calibration of subdistribution hazard models in discrete time

The calibration of initial shocks in bank stress test scenarios: An outlier detection based approach