Abstract:High-dimensional linear regression under heavy-tailed noise or outlier corruption is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since the robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a projected sub-gradient descent algorithm for both the sparse linear regression and low-rank linear regression problems. The algorithm is not only computationally efficient with linear convergence but also statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1 + epsilon moment. The convergence theory is established for a general framework and its specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of two-phase convergence. In phase one, the algorithm behaves as in typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator, which is already observed in the existing literature. Interestingly, during phase two, the algorithm converges linearly as if minimizing a smooth and strongly convex objective function, and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of our algorithm over prior methods.

A Rank-based Robust Linear Regression Analysis and Diagnostic Methods

Rank-based Robust Regression Method

THE METHOD OF RANK-BASED ROBUST REGRESSION ANALYSIS

A Novel Bayesian Robust Model and Its Application for Fault Detection and Automatic Supervision of Nonlinear Process

Residuals and Regression Diagnostics: Focusing on Logistic Regression

Robust reduced-rank regression

A Distributionally Robust Optimization Approach for Outlier Detection *

Low-Rank-Sparse Subspace Representation for Robust Regression

COMPARISONS BETWEEN ROBUST REGRESSION APPROACHES IN THE PRESENCE OF OUTLIERS AND HIGH LEVERAGE POINTS

A Robust Adaptive Modified Maximum Likelihood Estimator for the Linear Regression Model

Linear Regression Analysis of Censored Survival Data Based on Rank Tests

Outlier detection and robust variable selection via the penalized weighted LAD-LASSO method

A Robust Learning Approach for Regression Models Based on Distributionally Robust Optimization.

Correlation-Based Robust Linear Regression with Iterative Outlier Removal

Nonparametric Rank Test for Nonlinearity Detection

Rank Regression Analysis of Multivariate Failure Time Data Based on Marginal Linear Models

A Robust Learning Algorithm for Regression Models Using Distributionally Robust Optimization under the Wasserstein Metric

Efficient Estimation for Rank-Based Regression with Clustered Data

DETECTION OF MULTIPLE OUTLIERS BY RANDOM ROBUST TESTING

Robust Partial Least Squares Regression - Part Iii, Outlier Analysis and Application Studies

Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression