Abstract:Assessment of risk prediction models has primarily utilized measures of discrimination, the ROC curve AUC and C-statistic. These derive from the risk distributions of patients and nonpatients, which in turn are derived from a population risk distribution. As greater dispersion of the population risk distribution produces greater separation of patient and nonpatient risks (discrimination), its parameters can be used as alternatives to the ROC curve AUC and C-statistic. Here continuous probability distributions are employed to develop insight into the relationship between their parameters and the ROC curve AUC and C-statistic derived from them. The ROC curve AUC and C-statistic are shown to have a straight-line relationship with the SD for uniform, half-sine, and symmetric triangular probability distributions, with slight differences in the slope: AUC approx 1/2+0.28 SD/(mean(1-mean)). This also characterizes the beta distribution over the same range of SD's. But at larger beta distribution SD's the plot of AUC versus SD deviates downward from this straight-line relationship, approaching the ROC curve AUC and SD of a perfect model (AUC=1, SD= $\sqrt{\rm mean(1-mean)}$). A simpler and more intuitive discrimination metric is the coefficient of discrimination, the difference between the mean risk in patients and nonpatients. This is SD2/(mean(1-mean)), which is also the same for any distribution. Since estimating parameters or metrics discards information, the population risk distribution should always be presented. As the ROC curve AUC and C-statistic are functions of this distribution's parameters, the parameters represent simpler, intuitive alternatives to these discrimination metrics. Among discrimination metrics, the coefficient of discrimination provides a simple, intuitive alternative to the ROC curve AUC and C-statistic.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is some limitations of the traditional methods of relying on the area under the ROC curve (AUC) and the C - statistic as measures of discrimination ability when evaluating risk prediction models. The author believes that although these measures are widely used, they are not the most intuitive or the best choices to reflect model performance. ### Main problems: 1. **Limitations of discrimination measures**: Traditional AUC and C - statistic are mainly based on the distribution differences between patients and non - patients, while ignoring the more fundamental population risk distribution. This approach may lead to an incomplete understanding of model performance. 2. **Importance of population risk distribution**: The author emphasizes that the risk distributions of patients and non - patients are determined by the overall population risk distribution. Therefore, extracting parameters directly from the population risk distribution can provide a simpler and more intuitive alternative to evaluate the discrimination ability of the model. 3. **Exploring new measures**: The paper explores the possibility of using parameters of the population risk distribution (such as mean and standard deviation) to evaluate model performance and proposes several alternative indicators, for example: - **Coefficient of Discrimination**: Defined as the difference between the average risks of patients and non - patients, and the formula is: \[ \text{Coefficient of Discrimination}=\frac{\text{SD}^2}{\text{mean}(1 - \text{mean})} \] - **Brier score**: Used to measure the accuracy of prediction, and the formula is: \[ \text{Brier score}=\text{mean}(1 - \text{mean})-\text{SD}^2 \] - **Log - likelihood**: Used to measure the goodness - of - fit of the model, and has a similar curvature relationship for uniform distribution and beta distribution. 4. **Visual display**: The author suggests that when evaluating risk prediction models, the population risk distribution map (such as probability density function, cumulative risk distribution curve, etc.) should be preferentially displayed rather than relying solely on a single numerical indicator. ### Conclusion: The paper proposes that by directly analyzing and displaying the population risk distribution, a more intuitive and comprehensive evaluation method for risk prediction models can be provided. Compared with the traditional AUC and C - statistic, this method not only simplifies the calculation but also provides more information, which is helpful for better understanding the performance of the model and its clinical application value.

Alternatives to the ROC Curve AUC and C-statistic for Risk Prediction Models

Interpretation of the Area Under the ROC Curve for Risk Prediction Models

The Risk Distribution Curve and its Derivatives

Decision Curve Analysis: a Technical Note

Improving the Presentation and Understanding of Risk Models

Avoiding C-hacking when evaluating survival distribution predictions with discrimination measures

A comparison of confidence/credible interval methods for the area under the ROC curve for continuous diagnostic tests with small sample size

ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

Evaluating discriminatory accuracy of models using partial risk-scores in two-phase studies

Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification

A Marginal Model Approach for Analysis of Multi-Reader Multi-Test Receiver Operating Characteristic (ROC) Data

ROCnReg: An R Package for Receiver Operating Characteristic Curve Inference with and without Covariate Information

A Non-Parametric Method for the Comparison of Partial Areas under ROC Curves and Its Application to Large Health Care Data Sets.

Joint Statistical Inference for the Area under the ROC Curve and Youden Index under a Density Ratio Model

Methodological concerns about 'concordance-statistic for benefit' as a measure of discrimination in treatment benefit prediction

Nonparametric Covariate Adjustment for Receiver Operating Characteristic Curves

Scored AUC Metrics for Classifier Evaluation and Selection

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation-based methods

Confidence Intervals of the Difference Between Areas under Two ROC Curves in Matched‐pair Experiments

A robust approach for ROC curves with covariates

Comparison of methods for calculating confidence intervals of AUC in ROC curve considering sampling error