Abstract:In classification, the use of 0-1 loss is preferable since the minimizer of 0-1 risk leads to the Bayes optimal classifier. However, due to the nonconvexity of 0-1 loss, this optimization problem is NP-hard. Therefore, many convex surrogate loss functions have been adopted. Previous works have shown that if a Bayes-risk consistent loss function is used as a surrogate, the minimizer of the empirical surrogate risk can achieve the Bayes optimal classifier as the sample size tends to infinity. Nevertheless, the comparison of convergence rates of minimizers of different empirical surrogate risks to the Bayes optimal classifier has rarely been studied. Which characterization of the surrogate loss determines its convergence rate to the Bayes optimal classifier? Can we modify the loss function to achieve a faster convergence rate? In this article, we study the convergence rates of empirical surrogate minimizers to the Bayes optimal classifier. Specifically, we introduce the notions of consistency intensity and conductivity to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer to the Bayes optimal classifier, enabling fair comparisons of the excess risks of different surrogate risk minimizers. The main result of this article has practical implications including: 1) showing that hinge loss (SVM) is superior to logistic loss (Logistic regression) and exponential loss (Adaboost) in the sense that its empirical minimizer converges faster to the Bayes optimal classifier and 2) guiding the design of new loss functions to speed up the convergence rate to the Bayes optimal classifier with a data-dependent loss correction method inspired by our theorems.

Formal Definitions of Conservative Probability Distribution Functions (PDFs)

Combining information from independent sources through confidence distributions

Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)

Fusion of Probability Density Functions

Consistent fusion method with uncertainty elimination for distributed multi-sensor systems

Conservative Updating

Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds

A Statistical Theory for the Analysis of Uncertain Systems

Partial Consensus and Conservative Fusion of Gaussian Mixtures for Distributed PHD Fusion

Non-Exchangeable Conformal Risk Control

Covariance's Loss is Privacy's Gain: Computationally Efficient, Private and Accurate Synthetic Data

Conservative Prediction via Data-Driven Confidence Minimization

Some results on a class of functional optimization problems

Technical Communique: On the binomial confidence interval and probabilistic robust control

A Quarter Century of Covariance Intersection: Correlations Still Unknown?

Efficient Conformal Prediction under Data Heterogeneity

Wasserstein $F$-tests and Confidence Bands for the Frèchet Regression of Density Response Curves

Distributional stability of sparse inverse covariance matrix estimators

On the Rates of Convergence from Surrogate Risk Minimizers to the Bayes Optimal Classifier.

Strictly Frequentist Imprecise Probability

Conformal Inference for Invariant Risk Minimization