Abstract:We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. We design several robust algorithms that outperform the state of the art even in the special case when oblivious adversary simply adds Gaussian noise. In particular, we provide a polynomial-time algorithm that with high probability recovers the signal up to error $O(\sqrt{\varepsilon})$ as long as the number of samples $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of the distribution ${D}$ of the design.
In addition, prior to this work, even in the special case of Gaussian design and noise, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$. We show that under some assumptions on the fourth and the eighth moments of ${D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. For Gaussian distribution, this algorithm achieves error $O(\varepsilon^{3/4})$. Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$.
Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.
What problem does this paper attempt to address?
This paper attempts to solve the robust estimation problem in sparse linear regression when two adversarial factors, namely oblivious adversarial and adaptive adversarial, exist simultaneously. Specifically, the research objective is to estimate the sparse vector \(\beta^*\) under the following conditions:
1. **Model Setup**:
- The observed value \(y^* = X^* \beta^*+\eta\), where \(X^*\) is an \(n\times d\) random design matrix, \(\beta^*\in\mathbb{R}^d\) is a \(k\)-sparse vector (i.e., it has only \(k\) non - zero elements), and \(\eta\) is noise.
- The noise \(\eta\) is chosen by an oblivious adversary and is independent of \(X^*\), and it is required that at least \(0.01\cdot n\) elements have an absolute value not exceeding a certain constant \(\epsilon\).
2. **Adversarial Setting**:
- The adaptive adversary can arbitrarily tamper with \(\delta\) proportion of sample pairs \((X^*_i,y^*_i)\), that is, it can observe \(X^*, \beta^*, \eta\) and tamper according to this information.
3. **Objective**:
- After being given the tampered sample pairs \((X_i,y_i)\), estimate the original sparse vector \(\beta^*\).
4. **Assumptions**:
- The rows of the design matrix \(X^*\) are independent and identically distributed \(d\)-dimensional random vectors from a distribution \(\mathcal{D}\) with a mean of zero and a covariance matrix \(\Sigma\), and the condition number of \(\Sigma\) is bounded.
- For Gaussian noise \(\eta\sim\mathcal{N}(0, I_n)\), there is no previously known polynomial - time algorithm that can achieve an error of \(O(\sqrt{\delta})\) when \(n < d^2\).
5. **Main Contributions**:
- Several robust algorithms are proposed, which can surpass the existing state - of - the - art under specific conditions. For example, in the case of Gaussian noise, a polynomial - time algorithm is provided, which can recover \(\beta^*\) to an error of \(O(\sqrt{\delta})\) with high probability when \(n\gtrsim k^2/\delta\).
- For more general heavy - tailed distribution designs, a method based on filtering covariates and weighted Huber loss minimization is proposed, and new theoretical analysis is provided, which is applicable to scenarios where there are two adversaries.
In summary, this paper aims to solve the robust estimation problem in sparse linear regression when there are two different types of adversarial interference simultaneously, and proposes a series of effective algorithms and theoretical results.