Abstract:Performance metrics are measures of success or performance that can be used to evaluate how well a model makes accurate predictions or classifications. However, there is no single measure since each performance metric emphasizes a different classification aspect. Model selection procedures based on information criteria offer a quantitative measure that balances model complexity with goodness of fit, providing a better alternative to classical approaches. In this paper, we introduce and develop a novel Information Complexity–Receiver Operating Characteristic, abbreviated as ICOMP-ROC, criterion approach to fit and study the performance of ROC curve models. We construct and derive the Universal ROC (UROC) for a combination of sixteen Bi-distributional ROC models to choose the best Bi-distributional ROC by minimizing the ICOMP-ROC criterion. We conduct large-scale Monte Carlo simulations using the sixteen Bi-distributional ROC models with the Normal–Normal and Weibull–Gamma pairs as the pseudo-true ROC models. We report the frequency of hits of the ICOMP-ROC criterion, showing its remarkable recovery rate. In addition to Bi-distributional fitting, we consider a high-dimensional real Magnetic Resonance Imaging (MRI) of the Brain dataset and Wisconsin Breast Cancer (WBC) dataset to study the performance of the common performance metrics and the ICOMP-ROC criterion using several machine learning (ML) classification algorithms. We use the genetic algorithm (GA) to reduce the dimensions of these two datasets to choose the best subset of the features to study and compare the performance of the newly proposed ICOMP-ROC criterion along with the traditional performance metrics. The choice of a suitable metric is not just contingent upon the ML model used, but it also depends upon the complexity and high dimensionality of the input datasets, since the traditional performance metrics give different results and have inherent limitations. Our numerical results show the consistency and reliability of the ICOMP-ROC criterion over the traditional performance metrics as a clever model selection criterion to choose the best fitting Bi-distributional ROC model and the best classification algorithm among the ones considered. This shows the utility and the versatility of our newly proposed approach in ROC curve modeling that integrates and robustifies currently used procedures.

Decision Making with Machine Learning and ROC Curves

Decision Curve Analysis: a Technical Note

Understanding decision curve analysis in clinical prediction model research

Interpretation of the Area Under the ROC Curve for Risk Prediction Models

A placement-value based approach to concave ROC analysis

What is an ROC curve?

ROCnReg: An R Package for Receiver Operating Characteristic Curve Inference with and without Covariate Information

Getting the ROC into Sync

A Clustered Optimal ROC Curve Method for Family-Based Genetic Risk Prediction

A Novel Information Complexity Approach to Score Receiver Operating Characteristic (ROC) Curve Modeling

A Marginal Model Approach for Analysis of Multi-Reader Multi-Test Receiver Operating Characteristic (ROC) Data

Order-restricted inference for clustered ROC data with application to fingerprint matching accuracy

Bayesian nonparametric inference for the covariate-adjusted ROC curve

Impact of methodological assumptions and covariates on the cutoff estimation in ROC analysis

Receiver Operating Characteristic Curves and Confidence Bands for Support Vector Machines

Nonparametric ROC Summary Statistics for Correlated Diagnostic Marker Data

A Non-Parametric Method for the Comparison of Partial Areas under ROC Curves and Its Application to Large Health Care Data Sets.

Learning Curves for Decision Making in Supervised Machine Learning -- A Survey

ROC Curve Estimation under Test-Result-dependent Sampling.

A Nonparametric Approach for Partial Areas under ROC Curves and Ordinal Dominance Curves

A robust approach for ROC curves with covariates