Abstract:Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss, and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in the existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Lastly, RsGrad is applicable for low-rank tensor estimation under heavy-tailed noise where a statistically optimal rate is attainable with the same phenomenon of dual-phase convergence, and a novel shrinkage-based second-order moment method is guaranteed to deliver a warm initialization. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods.

Efficient and provable online reduced rank regression via online gradient descent

Provable accelerated gradient method for nonconvex low rank optimization

Efficient Nonconvex Optimization for Two-way Sparse Reduced-Rank Regression

Projected Gradient Descent Algorithm for Low-Rank Matrix Estimation

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

Stochastic Variance Reduced Gradient for Affine Rank Minimization Problem

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Robust reduced-rank regression

Online Tensor Inference

Efficient Online Learning for Large-Scale Sparse Kernel Logistic Regression

Fast Gradient Method for Low-Rank Matrix Estimation

Efficient Nonlinear Classification Via Low-Rank Regularised Least Squares.

On low-rank regularized least squares for scalable nonlinear classification

Accelerated Gradient Method for A Class of Nonconvex Low Rank Problem: Essentially Matching the Optimal Convex Convergence Rate

Fast and Strong Convergence of Online Learning Algorithms.

Greedy Learning of Generalized Low-Rank Models.

A randomized algorithm to solve reduced rank operator regression

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix and Tensor Estimation

Low-rank kernel regression with preserved locality for multi-class analysis

Online Inference with Debiased Stochastic Gradient Descent

Learning of Generalized Low-Rank Models: A Greedy Approach.