Abstract:The surge of interest in personalized and precision medicine during recent years has increased the application of ordinal classification problems in biomedical science. Currently, accuracy, Kendall's τb , and average mean absolute error are three commonly used metrics for evaluating the effectiveness of an ordinal classifier. Although there are benefits to each, no single metric considers the benefits of predictive accuracy with the tradeoffs of misclassification cost. In addition, decision analysis that considers pairwise analysis of the metrics is not trivial due to inconsistent findings. A new cost-sensitive metric is proposed to find the optimal tradeoff between the two most critical performance measures of a classification task - accuracy and cost. The proposed method accounts for an inherent ordinal data structure, total misclassification cost of a classifier, and imbalanced class distribution. The strengths of the new methodology are demonstrated through analyses of three real cancer datasets and four simulation studies. The new cost-sensitive metric proved better performance in its ability to identify the best ordinal classifier for a given analysis. The performance metric devised in this study provides a comprehensive tool for comparative analysis of multiple (and competing) ordinal classifiers. Consideration of the tradeoff between accuracy and misclassification cost in decisions regarding ordinal classification problems is imperative in real-world application. The work presented here is a precursor to the possibility of incorporating the proposed metric into a prediction modeling algorithm for ordinal data as a means of integrating misclassification cost in final model selection.

An Empirical Study of MAUC in Multi-class Problems with Uncertain Cost Matrices

Minimax Classifier for Uncertain Costs

A Multi-Class Large Margin Classifier

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

On Multi-Class Cost-Sensitive Learning

AUCμ: A Performance Metric for Multi-Class Machine Learning Models

Multiclass ROC

A Decision-Theoretic Rough Set Approach To Multi-Class Cost-Sensitive Classification

Optimization of Selective Ensemble for Cost-Sensitive Classification: an Empirical Study

Cost-Sensitive Alternating Direction Method Of Multipliers For Large-Scale Classification

Feature selection for MAUC-oriented classification systems

Cost-sensitive matrixized classification learning with information entropy

An Empirical Study Of Metacost Using Boosting Algorithms

SVM-Based Cost-sensitive Classification Algorithm with Error Cost and Class-dependent Reject Cost

Angle-Based Cost-Sensitive Multicategory Classification

Binary Classification Algorithm with Class-Dependent Reject Cost

Cost-sensitive performance metric for comparing multiple ordinal classifiers

Cost Sensitive Support Vector Machines

Cost-Sensitive Classification Using Decision Trees, Boosting and MetaCost

Towards cost-sensitive learning for real-world applications

A Closer Look at AUROC and AUPRC under Class Imbalance