Abstract:As one of the triumphs and milestones of robust statistics, Huber regression plays an important role in robust inference and estimation. It has also been finding a great variety of applications in machine learning. In a parametric setup, it has been extensively studied. However, in the statistical learning context where a function is typically learned in a nonparametric way, there is still a lack of theoretical understanding of how Huber regression estimators learn the conditional mean function and why it works in the absence of light-tailed noise assumptions. To address these fundamental questions, we conduct an assessment of Huber regression from a statistical learning viewpoint. First, we show that the usual risk consistency property of Huber regression estimators, which is usually pursued in machine learning, cannot guarantee their learnability in mean regression. Second, we argue that Huber regression should be implemented in an adaptive way to perform mean regression, implying that one needs to tune the scale parameter in accordance with the sample size and the moment condition of the noise. Third, with an adaptive choice of the scale parameter, we demonstrate that Huber regression estimators can be asymptotic mean regression calibrated under $(1+\epsilon)$-moment conditions ($\epsilon>0$). Last but not least, under the same moment conditions, we establish almost sure convergence rates for Huber regression estimators. Note that the $(1+\epsilon)$-moment conditions accommodate the special case where the response variable possesses infinite variance and so the established convergence rates justify the robustness feature of Huber regression estimators. In the above senses, the present study provides a systematic statistical learning assessment of Huber regression estimators and justifies their merits in terms of robustness from a theoretical viewpoint.

Support Vector Machines Regression with Unbounded Sampling

Online Regression with Unbounded Sampling.

M-estimator Based Support Vector Machine and Its Application

ERM Learning with Unbounded Sampling

Local Linear M‐estimation in Non‐parametric Spatial Regression

Iterative robust least square support vector machine for spectral analysis

Concentration Estimates for Learning with Unbounded Sampling

When can Regression-Adjusted Control Variates Help? Rare Events, Sobolev Embedding and Minimax Optimality

Using Support Vector Machines for Mining Regression Classes in Large Data Sets

Least squares regression under weak moment conditions

Local Uncertainty Sampling for Large-Scale Multi-Class Logistic Regression

Sparse Least Squares Support Vector Machine for Function Estimation

Least Square Generalized Support Vector Machines for Regression

Uncertain least square support vector regression with imprecise observations

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Spatial local M-estimation under association

Fuzzy Support Vector Machine For Regression Estimation

Optimal Subsampling Approaches for Large Sample Linear Regression

A Statistical Learning Assessment of Huber Regression

Error analysis of regularized trigonometric linear regression with unbounded sampling: a statistical learning viewpoint

Large sample behavior of the least trimmed squares estimator