Abstract:The performance of machine learning classification algorithms are evaluated by estimating metrics, often from the confusion matrix, using training data and cross-validation. However, these do not prove that the best possible performance has been achieved. Fundamental limits to error rates can be estimated using information distance measures. To this end, the confusion matrix has been formulated to comply with the Chernoff-Stein Lemma. This links the error rates to the Kullback-Leibler divergences between the probability density functions describing the two classes. This leads to a key result that relates Cohen's Kappa to the Resistor Average Distance which is the parallel resistor combination of the two Kullback-Leibler divergences. The Resistor Average Distance has units of bits and is estimated from the same training data used by the classification algorithm, using kNN estimates of the KullBack-Leibler divergences. The classification algorithm gives the confusion matrix and Kappa. Theory and methods are discussed in detail and then applied to Monte Carlo data and real datasets. Four very different real datasets - Breast Cancer, Coronary Heart Disease, Bankruptcy, and Particle Identification - are analysed, with both continuous and discrete values, and their classification performance compared to the expected theoretical limit. In all cases this analysis shows that the algorithms could not have performed any better due to the underlying probability density functions for the two classes. Important lessons are learnt on how to predict the performance of algorithms for imbalanced data using training datasets that are approximately balanced. Machine learning is very powerful but classification performance ultimately depends on the quality of the data and the relevance of the variables to the problem.

Refined Statistical Bounds for Classification Error Mismatches with Constrained Bayes Error

A Bound on Mean-Square Estimation Error with Background Parameter Mismatch

Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means

Improving Statistical Robustness of Matched-Field Source Localization Via General-Rank Covariance Matrix Matching

A Bilateral Bound on the Mean-Square Error for Estimation in Model Mismatch

Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound

Approximate Bayesian Computation via Classification

Limits to classification performance by relating Kullback-Leibler divergence to Cohen's Kappa

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Moments and Root-Mean-Square Error of the Bayesian MMSE Estimator of Classification Error in the Gaussian Model

Robust Bayesian Inference for Berkson and Classical Measurement Error Models

Sharp error bounds for imbalanced classification: how many examples in the minority class?

BSM loss: A superior way in modeling aleatory uncertainty of fine_grained classification

Conditional Confidence Intervals for the True Classification Error

Multiclass Classification, Information, Divergence and Surrogate Risk

Metrics for Measuring Error Extents of Machine Learning Classifiers

Misclassification excess risk bounds for PAC-Bayesian classification via convexified loss

Intrinsic Fairness-Accuracy Tradeoffs under Equalized Odds

Classification and Regression Error Bounds for Inhomogenous Data With Applications to Wireless Networks

Minimax-Optimal Bounds for Detectors Based on Estimated Prior Probabilities

Improved Error Bounds Based on Worst Likely Assignments