Abstract:We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. We design several robust algorithms that outperform the state of the art even in the special case when oblivious adversary simply adds Gaussian noise. In particular, we provide a polynomial-time algorithm that with high probability recovers the signal up to error $O(\sqrt{\varepsilon})$ as long as the number of samples $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of the distribution ${D}$ of the design. In addition, prior to this work, even in the special case of Gaussian design and noise, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$. We show that under some assumptions on the fourth and the eighth moments of ${D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. For Gaussian distribution, this algorithm achieves error $O(\varepsilon^{3/4})$. Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$. Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.

Stahel-Donoho Kernel Estimation for Fixed Design Nonparametric Regression Models

Change point estimators by local polynomial fits under a dependence assumption

Asymptotic Behavior for S-estimators in Random Design Linear Model with Long-Range-dependent Errors

Local Linear M‐estimation in Non‐parametric Spatial Regression

Robust Regularized Kernel Regression.

On the Estimation of Derivatives Using Plug-in Kernel Ridge Regression Estimators

Quick and Simple Kernel Differential Equation Regression Estimators for Data with Sparse Design

Central Limit Theorems of a Recursive Stochastic Algorithm with Applications to Adaptive Designs

Stochastic Low-Rank Kernel Learning for Regression

Universal kernel-type estimation of random fields

An RKHS-based approach to double-penalized regression in high-dimensional partially linear models

Multivariate Universal Local Linear Kernel Estimators in Nonparametric Regression: Uniform Consistency

Nonparametric predictive model for sparse and irregular longitudinal data

Nonparametric Estimation via Partial Derivatives

Semiparametric efficient estimation in high‐dimensional partial linear regression models

Robust Sparse Regression with Non-Isotropic Designs

Asymptotic Distributions of Nonparametric Regression Estimators for Longitudinal or Functional Data

Function Estimation Using Data Adaptive Kernel Estimation - How Much Smoothing?

Doubly Robust Semiparametric Inference Using Regularized Calibrated Estimation with High-dimensional Data

Efficient and fast spline-backfitted kernel smoothing of additive models

Nonparametric Estimation of Copula Functions for Dependence Modelling