Abstract:In Machine Learning, a supervised model's performance is measured using the evaluation metrics. In this study, we first present our motivation by revisiting the major limitations of these metrics, namely one-dimensionality, lack of context, lack of intuitiveness, uncomparability, binary restriction, and uncustomizability of metrics. In response, we propose Contingency Space, a bounded semimetric space that provides a generic representation for any performance evaluation metric. Then we showcase how this space addresses the limitations. In this space, each metric forms a surface using which we visually compare different evaluation metrics. Taking advantage of the fact that a metric's surface warps proportionally to the degree of which it is sensitive to the class-imbalance ratio of data, we introduce Imbalance Sensitivity that quantifies the skew-sensitivity. Since an arbitrary model is represented in this space by a single point, we introduce Learning Path for qualitative and quantitative analyses of the training process. Using the semimetric that contingency space is endowed with, we introduce Tau as a new cost sensitive and Imbalance Agnostic metric. Lastly, we show that contingency space addresses multi-class problems as well. Throughout this work, we define each concept through stipulated definitions and present every application with practical examples and visualizations.

Generalizing the Mean and Variance to Categorical Data Using Metrics

On mean and/or variance mixtures of normal distributions

A note on distance variance for categorical variables

We Need to Talk About Classification Evaluation Metrics in NLP

Supervised Categorical Metric Learning with Schatten p-Norms

Multivariate medians and measure-symmetrization

Generalization Bounds for Metric and Similarity Learning

Random matrix approach to multivariate categorical data analysis

MCMC for Imbalanced Categorical Data

On variational principles for metric mean dimension

Magnitude of arithmetic scalar and matrix categories

Easy Variational Inference for Categorical Models via an Independent Binary Approximation

An Effectiveness Metric for Ordinal Classification: Formal Properties and Experimental Results

Contingency Space: A Semimetric Space for Classification Evaluation

Generalized Approach for Using Unbiased Symmetric Metrics with Negative Values: Normalized Mean Bias Factor and Normalized Mean Absolute Error Factor

A Mean Field Games model for finite mixtures of Bernoulli and Categorical distributions

Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measure

Metric Space Magnitude for Evaluating the Diversity of Latent Representations

ANOVA for Data in Metric Spaces, with Applications to Spatial Point Patterns

Coalgebraic Behavioral Metrics

On Metrics for Probabilistic Systems: Definitions and Algorithms