Abstract:Binary classification is a common task for which machine learning and computational statistics are used, and the area under the receiver operating characteristic curve (ROC AUC) has become the common standard metric to evaluate binary classifications in most scientific fields. The ROC curve has true positive rate (also called sensitivity or recall ) on the y axis and false positive rate on the x axis, and the ROC AUC can range from 0 (worst result) to 1 (perfect result). The ROC AUC, however, has several flaws and drawbacks. This score is generated including predictions that obtained insufficient sensitivity and specificity, and moreover it does not say anything about positive predictive value (also known as precision ) nor negative predictive value (NPV) obtained by the classifier, therefore potentially generating inflated overoptimistic results. Since it is common to include ROC AUC alone without precision and negative predictive value, a researcher might erroneously conclude that their classification was successful. Furthermore, a given point in the ROC space does not identify a single confusion matrix nor a group of matrices sharing the same MCC value. Indeed, a given (sensitivity, specificity) pair can cover a broad MCC range, which casts doubts on the reliability of ROC AUC as a performance measure. In contrast, the Matthews correlation coefficient (MCC) generates a high score in its interval only if the classifier scored a high value for all the four basic rates of the confusion matrix: sensitivity, specificity, precision, and negative predictive value. A high MCC (for example, MCC 0.9), moreover, always corresponds to a high ROC AUC, and not vice versa. In this short study, we explain why the Matthews correlation coefficient should replace the ROC AUC as standard statistic in all the scientific studies involving a binary classification, in all scientific fields.

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

Multiclass ROC

A Marginal Model Approach for Analysis of Multi-Reader Multi-Test Receiver Operating Characteristic (ROC) Data

Towards Maximizing the Area Under the ROC Curve for Multi-Class Classification Problems.

A Non-Parametric Method for the Comparison of Partial Areas under ROC Curves and Its Application to Large Health Care Data Sets.

An efficient variance estimator of AUC and its applications to binary classification

Linearithmic and unbiased implementation of DeLong's algorithm for comparing the areas under correlated ROC curves

Implications of Imbalanced Datasets for Empirical ROC-AUC Estimation in Binary Classification Tasks

Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification

A test for comparing conditional ROC curves with multidimensional covariates

Nonparametric Covariate Adjustment for Receiver Operating Characteristic Curves

An Empirical Study of MAUC in Multi-class Problems with Uncertain Cost Matrices

A Closer Look at AUROC and AUPRC under Class Imbalance

Learning with Multiclass AUC: Theory and Algorithms

Evaluating performance of multiple Bayes classifier based on AUC method

Scored AUC Metrics for Classifier Evaluation and Selection

Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification

On Evaluating Multi-class Network Traffic Classifiers Based on AUC

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence Into Account

The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification