Abstract:In classification, the use of 0-1 loss is preferable since the minimizer of 0-1 risk leads to the Bayes optimal classifier. However, due to the nonconvexity of 0-1 loss, this optimization problem is NP-hard. Therefore, many convex surrogate loss functions have been adopted. Previous works have shown that if a Bayes-risk consistent loss function is used as a surrogate, the minimizer of the empirical surrogate risk can achieve the Bayes optimal classifier as the sample size tends to infinity. Nevertheless, the comparison of convergence rates of minimizers of different empirical surrogate risks to the Bayes optimal classifier has rarely been studied. Which characterization of the surrogate loss determines its convergence rate to the Bayes optimal classifier? Can we modify the loss function to achieve a faster convergence rate? In this article, we study the convergence rates of empirical surrogate minimizers to the Bayes optimal classifier. Specifically, we introduce the notions of consistency intensity and conductivity to characterize a surrogate loss function and exploit this notion to obtain the rate of convergence from an empirical surrogate risk minimizer to the Bayes optimal classifier, enabling fair comparisons of the excess risks of different surrogate risk minimizers. The main result of this article has practical implications including: 1) showing that hinge loss (SVM) is superior to logistic loss (Logistic regression) and exponential loss (Adaboost) in the sense that its empirical minimizer converges faster to the Bayes optimal classifier and 2) guiding the design of new loss functions to speed up the convergence rate to the Bayes optimal classifier with a data-dependent loss correction method inspired by our theorems.

The Superiority of the Bayes Linear Unbiased Minimum Variance Estimator under Balanced Loss Function

The Superiority of Bayes Linear Unbiased Minimum Variance Estimator with Respect to Ridge Estimator

Optimal Capital Allocation Based on Weighted-Mean-Variance Principle

Bayes estimator for the inequality-constrained regression model*

On the Optimality of the Simple Bayesian Classifier under Zero-One Loss

Study of the dynamic mean-variance portfolio optimization with regard to bankruptcy

On the Rates of Convergence from Surrogate Risk Minimizers to the Bayes Optimal Classifier.

Bayesian estimation for mean vector of multivariate normal distribution on the linear and nonlinear exponential balanced loss based on wavelet decomposition

Performance analysis and optimal selection of large mean-variance portfolios under estimation risk

Nonlinear Monte Carlo Method for Imbalanced Data Learning

A Bayesian Risk Approach to Data-driven Stochastic Optimization: Formulations and Asymptotics

BSM loss: A superior way in modeling aleatory uncertainty of fine_grained classification

Broad Minimax Probability Learning System and its Application in Regression Modeling

A Statistical Perspective on Linear Programs with Uncertain Parameters

Optimal convex $M$-estimation via score matching

Efficient Empirical Bayes prediction under check loss using Asymptotic Risk Estimates

Risk-Aware MMSE Estimation

Is optimum always optimal? A revisit of the mean‐variance method under nonlinear measures of dependence and non‐normal liquidity constraints

On the Liu estimation of Bell regression model in the presence of multicollinearity

A Bayesian Skew-heavy-tailed modelling for loss reserving

Bayesian and minimax estimators of loss