Abstract:Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1+εmoment. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods. Jian-Feng Cai’s research was partially supported by Hong Kong RGC Grant GRF 16310620 and GRF 16309219. Dong Xia’s research was partially supported by Hong Kong RGC Grant ECS 26302019, GRF 16303320 and GRF 16300121.

Structured Gradient Descent for Fast Robust Low-Rank Hankel Matrix Completion

Accelerating Ill-conditioned Hankel Matrix Recovery via Structured Newton-like Descent

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Exact Matrix Completion Based on Low Rank Hankel Structure in the Fourier Domain

Adaptive Stochastic Gradient Descent on the Grassmannian for Robust Low-Rank Subspace Recovery

Nonconvex Deterministic Matrix Completion by Projected Gradient Descent Methods

Fast and Robust Spectrally Sparse Signal Recovery: A Provable Non-Convex Approach via Robust Low-Rank Hankel Matrix Reconstruction

Nearly-optimal Robust Matrix Completion

A Nearly Unbiased Matrix Completion Approach.

Quasi-Hankel low-rank matrix completion: a convex relaxation

A Preconditioned Riemannian Gradient Descent Algorithm for Low-Rank Matrix Recovery

Robust Matrix Completion with Deterministic Sampling via Convex Optimization

Fast and Accurate Matrix Completion Via Truncated Nuclear Norm Regularization

Low rank matrix completion by alternating steepest descent methods

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix Estimation

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Fast Algorithms for Recovering a Corrupted Low-Rank Matrix

Projected Gradient Descent for Spectral Compressed Sensing via Symmetric Hankel Factorization

A multi-stage convex relaxation approach to noisy structured low-rank matrix recovery

Robust Matrix Completion via Maximum Correntropy Criterion and Half Quadratic Optimization