Abstract:Consider a regression problem where the learner is given a large collection of $d$-dimensional data points, but can only query a small subset of the real-valued labels. How many queries are needed to obtain a $1+\epsilon$ relative error approximation of the optimum? While this problem has been extensively studied for least squares regression, little is known for other losses. An important example is least absolute deviation regression ($\ell_1$ regression) which enjoys superior robustness to outliers compared to least squares. We develop a new framework for analyzing importance sampling methods in regression problems, which enables us to show that the query complexity of least absolute deviation regression is $\Theta(d/\epsilon^2)$ up to logarithmic factors. We further extend our techniques to show the first bounds on the query complexity for any $\ell_p$ loss with $p\in(1,2)$. As a key novelty in our analysis, we introduce the notion of robust uniform convergence, which is a new approximation guarantee for the empirical loss. While it is inspired by uniform convergence in statistical learning, our approach additionally incorporates a correction term to avoid unnecessary variance due to outliers. This can be viewed as a new connection between statistical learning theory and variance reduction techniques in stochastic optimization, which should be of independent interest.

Strong convergence rate of the least median absolute estimator in linear regression models

A Least Squares Estimator for Lévy-driven Moving Averages Based on Discrete Time Observations

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Asymptotic normality for L1-norm kernel estimator of conditional median under association dependence

Local linear M-estimation for spatial processes in fixed-design models

LOCAL MEDIAN ESTIMATION OF VARIANCE FUNCTION

Strong convergence rates of several estimators in semiparametric varying-coefficient partially linear models

Fused LASSO Penalized Least Absolute Deviation Estimator for High Dimensional Linear Regression

Least Absolute Relative Error Estimation

On the convergence analysis of the transform domain normalized LMS and related M-estimate algorithms

Rate of Convergence of Normal Approximation for Local Median Estimate

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

Strong convergence rate of estimators of change point and its application

Analysis of Least Absolute Deviation

Optimal and Superoptimal Convergence Rate of the Local Linear Estimator of Nonparametric Regression Function in Continuous Time

A new penalized least absolute deviation model for high dimensional sparse linear regression and an efficient sequential linear programming algorithm

Error bounds of constant gain least-mean-squares algorithms

Asymptotics of Estimates in Constrained Nonlinear Regression with Long-Range Dependent Innovations

Regularized Least-Mean-Square Algorithms

Query Complexity of Least Absolute Deviation Regression via Robust Uniform Convergence

A new convergence rate of the steepest descent regarding the Euclidean norm