Abstract:High-dimensional linear regression under heavy-tailed noise or outlier corruption is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since the robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a projected sub-gradient descent algorithm for both the sparse linear regression and low-rank linear regression problems. The algorithm is not only computationally efficient with linear convergence but also statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1 + epsilon moment. The convergence theory is established for a general framework and its specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of two-phase convergence. In phase one, the algorithm behaves as in typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator, which is already observed in the existing literature. Interestingly, during phase two, the algorithm converges linearly as if minimizing a smooth and strongly convex objective function, and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of our algorithm over prior methods.

Semismooth Newton Coordinate Descent Algorithm for Elastic-Net Penalized Huber Loss Regression and Quantile Regression

Expectile regression for analyzing heteroscedasticity in high dimension

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Semismooth Newton Algorithm for High-Dimensional Nonconvex Sparse Learning

Semi-Smooth Newton Algorithm for Non-Convex Penalized Linear Regression

Hybrid Coordinate Descent for Efficient Neural Network Learning Using Line Search and Gradient Descent

High-Dimensional Composite Quantile Regression: Optimal Statistical Guarantees and Fast Algorithms

Generalized Newton–Raphson Algorithm for High Dimensional LASSO Regression

Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression

Coordinate Descent for MCP/SCAD Penalized Least Squares Converges Linearly

Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

A Fast Coordinate Descent Method for High-Dimensional Non-Negative Least Squares using a Unified Sparse Regression Framework

Decentralized Smoothing ADMM for Quantile Regression with Non-Convex Sparse Penalties

An Efficient Semi-smooth Newton Augmented Lagrangian Method for Elastic Net

SNAP: A semismooth Newton algorithm for pathwise optimization with optimal local convergence rate and oracle properties

Penalized weighted smoothed quantile regression for high‐dimensional longitudinal data

Iterative Reweighted Framework Based Algorithms for Sparse Linear Regression with Generalized Elastic Net Penalty

A regularized Newton method for $\ell_q$-norm composite optimization problems

Online Quantile Regression

Coordinate Descent Algorithm for Ramp Loss Linear Programming Support Vector Machines