Abstract:In classification, the use of 0-1 loss is preferable since the minimizer of 0-1 risk leads to the Bayes optimal classifier. However, due to the nonconvexity of 0-1 loss, this optimization problem is NP-hard. Therefore, many convex surrogate loss functions have been adopted. Previous works have shown that if a Bayes-risk consistent loss function is used as a surrogate, the minimizer of the empirical surrogate risk can achieve the Bayes optimal classifier as the sample size tends to infinity. Nevertheless, the comparison of convergence rates of minimizers of different empirical surrogate risks to the Bayes optimal classifier has rarely been studied. Which characterization of the surrogate loss determines its convergence rate to the Bayes optimal classifier? Can we modify the loss function to achieve a faster convergence rate? In this article, we study the convergence rates of empirical surrogate minimizers to the Bayes optimal classifier. Specifically, we introduce the notions of consistency intensity and conductivity to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer to the Bayes optimal classifier, enabling fair comparisons of the excess risks of different surrogate risk minimizers. The main result of this article has practical implications including: 1) showing that hinge loss (SVM) is superior to logistic loss (Logistic regression) and exponential loss (Adaboost) in the sense that its empirical minimizer converges faster to the Bayes optimal classifier and 2) guiding the design of new loss functions to speed up the convergence rate to the Bayes optimal classifier with a data-dependent loss correction method inspired by our theorems.

Optimal Convergence Rate of the Universal Estimation Error

A Supplement to the Complete Convergence

Rate Of Convergence For The Maximum Likelihood Estimator In Fractional Ornstein-Uhlenbeck Processes

On the Rates of Convergence from Surrogate Risk Minimizers to the Bayes Optimal Classifier.

Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory

The Rate of Convergence of AdaBoost

Convergence Rates of the Regularized Optimal Transport : Disentangling Suboptimality and Entropy

Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing

On the Optimal Rate for the Convergence Problem in Mean Field Control

Convergence Rates of Spectral Distributions of Large Sample Covariance Matrices

Optimal Estimation and Control for Lossy Network: Stability, Convergence, and Performance

Convergence Rates for the Maximum A Posteriori Estimator in PDE-Regression Models with Random Design

Fast Rate Generalization Error Bounds: Variations on a Theme.

Beyond Catoni: Sharper Rates for Heavy-Tailed and Robust Mean Estimation

Statistical Convergence Rates of Optimal Transport Map Estimation between General Distributions

Convergence rate of entropy-regularized multi-marginal optimal transport costs

On the Concentration of the Minimizers of Empirical Risks

Wasserstein convergence rates of increasingly concentrating probability measures

Convergence of Empirical Optimal Transport in Unbounded Settings

Universal Rates of Empirical Risk Minimization