Abstract:Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss, and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in the existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Lastly, RsGrad is applicable for low-rank tensor estimation under heavy-tailed noise where a statistically optimal rate is attainable with the same phenomenon of dual-phase convergence, and a novel shrinkage-based second-order moment method is guaranteed to deliver a warm initialization. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods.

Adaptive Stochastic Gradient Descent on the Grassmannian for Robust Low-Rank Subspace Recovery

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Low Rank Matrix Recovery with Adversarial Sparse Noise

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization

Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

Online Robust Subspace Tracking from Partial Information

Nonconvex Low-Rank Matrix Recovery with Arbitrary Outliers via Median-Truncated Gradient Descent

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix and Tensor Estimation

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix Estimation

Accelerated Stochastic Variance Reduction Gradient Algorithms for Robust Subspace Clustering

Iterative Online Subspace Learning for Robust Image Alignment

Structured Gradient Descent for Fast Robust Low-Rank Hankel Matrix Completion

Gradient Descent for Robust Kernel-Based Regression

A Unified Variance Reduction-Based Framework for Nonconvex Low-Rank Matrix Recovery

A Single-Mode Quasi Riemannian Gradient Descent Algorithm for Low-Rank Tensor Recovery

Outlier-robust sparse/low-rank least-squares regression and robust matrix completion

Incremental Gradient on the Grassmannian for Online Foreground and Background Separation in Subsampled Video

Robust Subspace Discovery Via Relaxed Rank Minimization

Efficient Low-Rank Semidefinite Programming With Robust Loss Functions

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems