Abstract:A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced accuracy, standard and balanced error rate, F-beta score, and Matthews correlation coefficient (MCC). In this document, we review the definition of these and other metrics and compare them with the expected cost (EC), a metric introduced in every statistical learning course but rarely used in the machine learning literature. We show that both the standard and balanced error rates are special cases of the EC. Further, we show its relation with F-beta score and MCC and argue that EC is superior to these traditional metrics for being based on first principles from statistics, and for being more general, interpretable, and adaptable to any application scenario. The metrics mentioned above measure the quality of hard decisions. Yet, most modern classification systems output continuous scores for the classes which we may want to evaluate directly. Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk, among others. The last three metrics are special cases of a family of metrics given by the expected value of proper scoring rules (PSRs). We review the theory behind these metrics, showing that they are a principled way to measure the quality of the posterior probabilities produced by a system. Finally, we show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE), arguing that calibration loss based on PSRs is superior to the ECE for being more interpretable, more general, and directly applicable to the multi-class case, among other reasons.

Unified Performance Measure for Binary Classification Problems

A unifying view for performance measures in multi-class prediction

A Unified View of Multi-Label Performance Measures.

Fair evaluation of classifier predictive performance based on binary confusion matrix

A Minimax Probability Machine for Nondecomposable Performance Measures

Empirical analysis of performance assessment for imbalanced classification

Cost-sensitive performance metric for comparing multiple ordinal classifiers

Analysis and Comparison of Classification Metrics

A study on cost behaviors of binary classification measures in class-imbalanced problems

BenchMetrics: a systematic benchmarking method for binary classification performance metrics

Testing the Consistency of Performance Scores Reported for Binary Classification Problems

Performance measures for classification systems with rejection

AUCμ: A Performance Metric for Multi-Class Machine Learning Models

Selecting a classification performance measure: matching the measure to the problem

Combined Bivariate Performance Measure

Robust performance metrics for imbalanced classification problems

A Robust profit measure for binary classification model evaluation

BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary classification problems

Exploring Symmetry of Binary Classification Performance Metrics