Abstract:Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss, and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in the existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Lastly, RsGrad is applicable for low-rank tensor estimation under heavy-tailed noise where a statistically optimal rate is attainable with the same phenomenon of dual-phase convergence, and a novel shrinkage-based second-order moment method is guaranteed to deliver a warm initialization. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods.

EGRank: an Exponentiated Gradient Algorithm for Sparse Learning-to-rank

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Greedy Learning of Generalized Low-Rank Models.

Learning of Generalized Low-Rank Models: A Greedy Approach.

Generalized Low-rank plus Sparse Tensor Estimation by Fast Riemannian Optimization

A Two-Stage Approach for Learning A Sparse Model with Sharp Excess Risk Analysis

Provable accelerated gradient method for nonconvex low rank optimization

Projected Gradient Descent Algorithm for Low-Rank Matrix Estimation

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

Low-Rank Extragradient Methods for Scalable Semidefinite Optimization

Sparse Learning for Large-Scale and High-Dimensional Data: A Randomized Convex-Concave Optimization Approach

Fast Gradient Method for Low-Rank Matrix Estimation

InRank: Incremental Low-Rank Learning

Fast Low-Rank Matrix Learning with Nonconvex Regularization

FSMRank: feature selection algorithm for learning to rank.

Accelerated Sparse Linear Regression Via Random Projection.

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix and Tensor Estimation

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Elastic Gradient Descent, an Iterative Optimization Method Approximating the Solution Paths of the Elastic Net

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Sparse and Low-Rank High-Order Tensor Regression via Parallel Proximal Method