Abstract:We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. We design several robust algorithms that outperform the state of the art even in the special case when oblivious adversary simply adds Gaussian noise. In particular, we provide a polynomial-time algorithm that with high probability recovers the signal up to error $O(\sqrt{\varepsilon})$ as long as the number of samples $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of the distribution ${D}$ of the design. In addition, prior to this work, even in the special case of Gaussian design and noise, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$. We show that under some assumptions on the fourth and the eighth moments of ${D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. For Gaussian distribution, this algorithm achieves error $O(\varepsilon^{3/4})$. Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$. Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.

Sparse Covariance and Precision Random Design Regression

Adaptive posterior concentration rates for sparse high-dimensional linear regression with random design and unknown error variance

Regularized Matrix Regression

A Constructive Approach to High-dimensional Regression

Robust Sparse Regression with Non-Isotropic Designs

Sparse estimation of a covariance matrix

A Constrained L1 Minimization Approach to Sparse Precision Matrix Estimation

Random design analysis of ridge regression

Estimation of Sparse Covariance Matrix Via Non-Convex Regularization

Adjusting for High-Dimensional Covariates in Sparse Precision Matrix Estimation by Lasso Penalized D-Trace Loss

Sparse Learning for Large-Scale and High-Dimensional Data: A Randomized Convex-Concave Optimization Approach

Penalized Sparse Covariance Regression with High Dimensional Covariates

High Dimensional Robust Sparse Regression

Graph-based regularization for regression problems with alignment and highly-correlated designs

Feature Adaptation for Sparse Linear Regression

Covariance-regularized regression and classification for high dimensional problems

Estimation of Error Variance Via Ridge Regression

Sparse nonlinear regression: Parameter estimation and asymptotic inference

Sparse Data-Driven Random Projection in Regression for High-Dimensional Data

A Unified Dynamic Approach to Sparse Model Selection