Abstract:A variety of different performance metrics are commonly used in the machine learning literature for the evaluation of classification systems. Some of the most common ones for measuring quality of hard decisions are standard and balanced accuracy, standard and balanced error rate, F-beta score, and Matthews correlation coefficient (MCC). In this document, we review the definition of these and other metrics and compare them with the expected cost (EC), a metric introduced in every statistical learning course but rarely used in the machine learning literature. We show that both the standard and balanced error rates are special cases of the EC. Further, we show its relation with F-beta score and MCC and argue that EC is superior to these traditional metrics for being based on first principles from statistics, and for being more general, interpretable, and adaptable to any application scenario. The metrics mentioned above measure the quality of hard decisions. Yet, most modern classification systems output continuous scores for the classes which we may want to evaluate directly. Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk, among others. The last three metrics are special cases of a family of metrics given by the expected value of proper scoring rules (PSRs). We review the theory behind these metrics, showing that they are a principled way to measure the quality of the posterior probabilities produced by a system. Finally, we show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE), arguing that calibration loss based on PSRs is superior to the ECE for being more interpretable, more general, and directly applicable to the multi-class case, among other reasons.

Contingency Space: A Semimetric Space for Classification Evaluation

Measuring Class-Imbalance Sensitivity of Deterministic Performance Evaluation Metrics

Empirical analysis of performance assessment for imbalanced classification

Never mind the metrics -- what about the uncertainty? Visualising confusion matrix metric distributions

Metric Learning Via Penalized Optimization.

On the Sampling Strategy for Evaluation of Spectral-Spatial Methods in Hyperspectral Image Classification

Iterative Metric Learning for Imbalance Data Classification

BenchMetrics: a systematic benchmarking method for binary classification performance metrics

Analysis and Comparison of Classification Metrics

Developing a Dataset-Adaptive, Normalized Metric for Machine Learning Model Assessment: Integrating Size, Complexity, and Class Imbalance

Machine Learning Capability: A standardized metric using case difficulty with applications to individualized deployment of supervised machine learning

Metric Space Magnitude for Evaluating the Diversity of Latent Representations

Cost-sensitive performance metric for comparing multiple ordinal classifiers

A New Similarity Space Tailored for Supervised Deep Metric Learning

Instance Specific Metric Subspace Learning: A Bayesian Approach.

An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results

Analysis of multi-class classification performance metrics for remote sensing imagery imbalanced datasets

Inducing Semantic Hierarchy Structure in Empirical Risk Minimization with Optimal Transport Measures

We Need to Talk About Classification Evaluation Metrics in NLP

Semi-supervised Sparse Metric Learning Using Alternating Linearization Optimization