Abstract:A large body of work in the statistics and computer science communities dating back to Huber (Huber, 1960) has led to statistically and computationally efficient outlier-robust estimators. Two particular outlier models have received significant attention: the adversarial and heavy-tailed models. While the former models outliers as the result of a malicious adversary manipulating the data, the latter relaxes distributional assumptions on the data allowing outliers to naturally occur as part of the data generating process. In the first setting, the goal is to develop estimators robust to the largest fraction of outliers while in the second, one seeks estimators to combat the loss of statistical efficiency, where the dependence on the failure probability is paramount. Despite these distinct motivations, the algorithmic approaches to both these settings have converged, prompting questions on the relationship between the models. In this paper, we investigate and provide a principled explanation for this phenomenon. First, we prove that any adversarially robust estimator is also resilient to heavy-tailed outliers for any statistical estimation problem with i.i.d data. As a corollary, optimal adversarially robust estimators for mean estimation, linear regression, and covariance estimation are also optimal heavy-tailed estimators. Conversely, for arguably the simplest high-dimensional estimation task of mean estimation, we construct heavy-tailed estimators whose application to the adversarial setting requires any black-box reduction to remove almost all the outliers in the data. Taken together, our results imply that heavy-tailed estimation is likely easier than adversarially robust estimation opening the door to novel algorithmic approaches for the heavy-tailed setting. Additionally, confidence intervals obtained for adversarially robust estimation also hold with high-probability.

Robust Regression with Covariate Filtering: Heavy Tails and Adversarial Contamination

Robust Estimation and Shrinkage in Ultrahigh Dimensional Expectile Regression with Heavy Tails and Variance Heterogeneity

Heavy-tailed Contamination is Easier than Adversarial Contamination

Universal Robust Regression via Maximum Mean Discrepancy

High-dimensional robust regression under heavy-tailed data: Asymptotics and Universality

Sparse Linear Regression when Noises and Covariates are Heavy-Tailed and Contaminated by Outliers

Outlier Robust and Sparse Estimation of Linear Regression Coefficients

Adversarial robust weighted Huber regression

Optimal Robust Estimation under Local and Global Corruptions: Stronger Adversary and Smaller Error

Tractable and Near-Optimal Adversarial Algorithms for Robust Estimation in Contaminated Gaussian Models

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Robust estimation with Lasso when outputs are adversarially contaminated

Posterior Robustness with Milder Conditions: Contamination Models Revisited

Robust Estimation and Inference for Categorical Data

Near-Optimal Algorithms for Gaussians with Huber Contamination: Mean Estimation and Linear Regression

A Robust Learning Approach for Regression Models Based on Distributionally Robust Optimization.

A Robust Learning Algorithm for Regression Models Using Distributionally Robust Optimization under the Wasserstein Metric

Robust Sparse Estimation for Gaussians with Optimal Error under Huber Contamination

Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails

Robust Regression with Compositional Covariates

Robust W-GAN-Based Estimation Under Wasserstein Contamination