Abstract:Bayesian classification labels observations based on given prior information, namely class-a priori and class-conditional probabilities. Bayes' risk is the minimum expected classification cost that is achieved by the Bayes' test, the optimal decision rule. When no cost incurs for correct classification and unit cost is charged for misclassification, Bayes' test reduces to the maximum a posteriori decision rule, and Bayes risk simplifies to Bayes' error, the probability of error. Since calculating this probability of error is often intractable, several techniques have been devised to bound it with closed-form formula, introducing thereby measures of similarity and divergence between distributions like the Bhattacharyya coefficient and its associated Bhattacharyya distance. The Bhattacharyya upper bound can further be tightened using the Chernoff information that relies on the notion of best error exponent. In this paper, we first express Bayes' risk using the total variation distance on scaled distributions. We then elucidate and extend the Bhattacharyya and the Chernoff upper bound mechanisms using generalized weighted means. We provide as a byproduct novel notions of statistical divergences and affinity coefficients. We illustrate our technique by deriving new upper bounds for the univariate Cauchy and the multivariate $t$-distributions, and show experimentally that those bounds are not too distant to the computationally intractable Bayes' error.

PAC-Bayesian Generalization Bound on Confusion Matrix for Multi-Class Classification

Confusion Matrix Stability Bounds for Multiclass Classification

Max-margin Based Bayesian Classifier.

Misclassification excess risk bounds for PAC-Bayesian classification via convexified loss

Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures

Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound

On the generalization of bayesian deep nets for multi-class classification

Improving Generalization of Complex Models under Unbounded Loss Using PAC-Bayes Bounds

PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons

PAC-Bayesian Theory Meets Bayesian Inference

Fast-Rate PAC-Bayesian Generalization Bounds for Meta-Learning.

Misclassification bounds for PAC-Bayesian sparse deep learning

PAC-Bayes-Chernoff bounds for unbounded losses

PAC-Bayes Analysis of Multi-view Learning

A General Framework for the Practical Disintegration of PAC-Bayesian Bounds

Better-than-KL PAC-Bayes Bounds

PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

A New Family of Generalization Bounds Using Samplewise Evaluated CMI

Tighter Generalisation Bounds via Interpolation

A PAC-Bayes Analysis of Adversarial Robustness

Generalized Bhattacharyya and Chernoff upper bounds on Bayes error using quasi-arithmetic means