Abstract:Performance metrics are measures of success or performance that can be used to evaluate how well a model makes accurate predictions or classifications. However, there is no single measure since each performance metric emphasizes a different classification aspect. Model selection procedures based on information criteria offer a quantitative measure that balances model complexity with goodness of fit, providing a better alternative to classical approaches. In this paper, we introduce and develop a novel Information Complexity–Receiver Operating Characteristic, abbreviated as ICOMP-ROC, criterion approach to fit and study the performance of ROC curve models. We construct and derive the Universal ROC (UROC) for a combination of sixteen Bi-distributional ROC models to choose the best Bi-distributional ROC by minimizing the ICOMP-ROC criterion. We conduct large-scale Monte Carlo simulations using the sixteen Bi-distributional ROC models with the Normal–Normal and Weibull–Gamma pairs as the pseudo-true ROC models. We report the frequency of hits of the ICOMP-ROC criterion, showing its remarkable recovery rate. In addition to Bi-distributional fitting, we consider a high-dimensional real Magnetic Resonance Imaging (MRI) of the Brain dataset and Wisconsin Breast Cancer (WBC) dataset to study the performance of the common performance metrics and the ICOMP-ROC criterion using several machine learning (ML) classification algorithms. We use the genetic algorithm (GA) to reduce the dimensions of these two datasets to choose the best subset of the features to study and compare the performance of the newly proposed ICOMP-ROC criterion along with the traditional performance metrics. The choice of a suitable metric is not just contingent upon the ML model used, but it also depends upon the complexity and high dimensionality of the input datasets, since the traditional performance metrics give different results and have inherent limitations. Our numerical results show the consistency and reliability of the ICOMP-ROC criterion over the traditional performance metrics as a clever model selection criterion to choose the best fitting Bi-distributional ROC model and the best classification algorithm among the ones considered. This shows the utility and the versatility of our newly proposed approach in ROC curve modeling that integrates and robustifies currently used procedures.

Information Geometric Model Selection Criterion and Its Application in Cognition

Support Vector Machines Based on Information Geometry

Matching 3D Models with Global Geometric Feature Map

The Best Fit Bayesian Hierarchical Generalized Linear Model Selection Using Information Complexity Criteria in the MCMC Approach

Large-Scale Model Selection with Misspecification

Choosing models in model-based clustering and discriminant analysis

Model Selection with Misspecified Spatial Covariance Structure

An Effective Model Selection Criterion for Mixtures of Gaussian Processes.

Image classification and cognition using contour curvature statistics

The Focused Information Criterion for Stochastic Model Selection Problems Using $M$-Estimators

Visual information quantification for object recognition and retrieval

Image cognition using contour curvature statistics

A Novel Information Complexity Approach to Score Receiver Operating Characteristic (ROC) Curve Modeling

Group Maximum Differentiation Competition: Model Comparison with Few Samples.

A Group Decision-Making Model with Interval Multiplicative Reciprocal Matrices Based on the Geometric Consistency Index

Classifier Geometrical Characteristic Comparison and Its Application in Classifier Selection

Information Geometry Theoretic Measures for Characterizing Neural Information Processing from Simulated EEG Signals

Sparse and geometry-aware generalisation of the mutual information for joint discriminative clustering and feature selection

When a good fit can be bad

Modeling Magnitude Discrimination: Effects of Internal Precision and Attentional Weighting of Feature Dimensions