Abstract:High-dimensional linear regression under heavy-tailed noise or outlier corruption is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since the robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a projected sub-gradient descent algorithm for both the sparse linear regression and low-rank linear regression problems. The algorithm is not only computationally efficient with linear convergence but also statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1 + epsilon moment. The convergence theory is established for a general framework and its specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of two-phase convergence. In phase one, the algorithm behaves as in typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator, which is already observed in the existing literature. Interestingly, during phase two, the algorithm converges linearly as if minimizing a smooth and strongly convex objective function, and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of our algorithm over prior methods.

Best Subset Selection for High-Dimensional Non-Smooth Models Using Iterative Hard Thresholding.

An Improved Algorithm for High-Dimensional Continuous Threshold Expectile Model with Variance Heterogeneity

Adaptive Iterative Hard Thresholding for Least Absolute Deviation Problems with Sparsity Constraints

Dual Iterative Hard Thresholding: from Non-convex Sparse Minimization to Non-smooth Concave Maximization.

Accelerated Iterative Hard Thresholding Algorithm for $$l_0$$ Regularized Regression Problem

Smoothing fast iterative hard thresholding algorithm for $\ell_0$ regularized nonsmooth convex regression problem

Dual Iterative Hard Thresholding.

Scaled minimax optimality in high-dimensional linear regression: A non-convex algorithmic regularization approach

A minimax optimal approach to high-dimensional double sparse linear regression

Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization.

A New Linear Convergence Result for the Iterative Soft Thresholding Algorithm

Probabilistic Iterative Hard Thresholding for Sparse Learning

Gradient hard thresholding pursuit for sparsity-constrained optimization

Computationally Efficient and Statistically Optimal Robust High-Dimensional Linear Regression

Dynamic Incremental Optimization for Best Subset Selection

A Smoothing Iterative Method for Quantile Regression with Nonconvex \begin{document}$ \ell_p $\end{document} Penalty

Gradient hard thresholding pursuit

Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space

Optimal $k$-thresholding algorithms for sparse optimization problems

Efficient Stochastic Gradient Hard Thresholding

Exact Recovery of Hard Thresholding Pursuit.