Abstract:Performance metrics are measures of success or performance that can be used to evaluate how well a model makes accurate predictions or classifications. However, there is no single measure since each performance metric emphasizes a different classification aspect. Model selection procedures based on information criteria offer a quantitative measure that balances model complexity with goodness of fit, providing a better alternative to classical approaches. In this paper, we introduce and develop a novel Information Complexity–Receiver Operating Characteristic, abbreviated as ICOMP-ROC, criterion approach to fit and study the performance of ROC curve models. We construct and derive the Universal ROC (UROC) for a combination of sixteen Bi-distributional ROC models to choose the best Bi-distributional ROC by minimizing the ICOMP-ROC criterion. We conduct large-scale Monte Carlo simulations using the sixteen Bi-distributional ROC models with the Normal–Normal and Weibull–Gamma pairs as the pseudo-true ROC models. We report the frequency of hits of the ICOMP-ROC criterion, showing its remarkable recovery rate. In addition to Bi-distributional fitting, we consider a high-dimensional real Magnetic Resonance Imaging (MRI) of the Brain dataset and Wisconsin Breast Cancer (WBC) dataset to study the performance of the common performance metrics and the ICOMP-ROC criterion using several machine learning (ML) classification algorithms. We use the genetic algorithm (GA) to reduce the dimensions of these two datasets to choose the best subset of the features to study and compare the performance of the newly proposed ICOMP-ROC criterion along with the traditional performance metrics. The choice of a suitable metric is not just contingent upon the ML model used, but it also depends upon the complexity and high dimensionality of the input datasets, since the traditional performance metrics give different results and have inherent limitations. Our numerical results show the consistency and reliability of the ICOMP-ROC criterion over the traditional performance metrics as a clever model selection criterion to choose the best fitting Bi-distributional ROC model and the best classification algorithm among the ones considered. This shows the utility and the versatility of our newly proposed approach in ROC curve modeling that integrates and robustifies currently used procedures.

Matrix variate receiver operating characteristic curve for binary classification

The receiver operating characteristic curve accurately assesses imbalanced datasets

A Marginal Model Approach for Analysis of Multi-Reader Multi-Test Receiver Operating Characteristic (ROC) Data

Nonparametric Covariate Adjustment for Receiver Operating Characteristic Curves

Receiver Operating Characteristic Curves and Confidence Bands for Support Vector Machines

ROCnReg: An R Package for Receiver Operating Characteristic Curve Inference with and without Covariate Information

A test for comparing conditional ROC curves with multidimensional covariates

A Novel Information Complexity Approach to Score Receiver Operating Characteristic (ROC) Curve Modeling

Statistical Inference for Box–Cox based Receiver Operating Characteristic Curves

Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification

Interpretation of the Area Under the ROC Curve for Risk Prediction Models

A robust approach for ROC curves with covariates

A placement-value based approach to concave ROC analysis

Confidence Bands for ROC Curves with Serially Dependent Data

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence Into Account

The Area Under the ROC Curve as a Measure of Clustering Quality

Multiclass ROC

Nonparametric statistical inference method for partial areas under receiver operating characteristic curves, with application to genomic studies.