Abstract:Low-rank matrix estimation under heavy-tailed noise is challenging, both computationally and statistically. Convex approaches have been proven statistically optimal but suffer from high computational costs, especially since robust loss functions are usually non-smooth. More recently, computationally fast non-convex approaches via sub-gradient descent are proposed, which, unfortunately, fail to deliver a statistically consistent estimator even under sub-Gaussian noise. In this paper, we introduce a novel Riemannian sub-gradient (RsGrad) algorithm which is not only computationally efficient with linear convergence but also is statistically optimal, be the noise Gaussian or heavy-tailed with a finite 1+εmoment. Convergence theory is established for a general framework and specific applications to absolute loss, Huber loss and quantile loss are investigated. Compared with existing non-convex methods, ours reveals a surprising phenomenon of dual-phase convergence. In phase one, RsGrad behaves as in a typical non-smooth optimization that requires gradually decaying stepsizes. However, phase one only delivers a statistically sub-optimal estimator which is already observed in existing literature. Interestingly, during phase two, RsGrad converges linearly as if minimizing a smooth and strongly convex objective function and thus a constant stepsize suffices. Underlying the phase-two convergence is the smoothing effect of random noise to the non-smooth robust losses in an area close but not too close to the truth. Numerical simulations confirm our theoretical discovery and showcase the superiority of RsGrad over prior methods. Jian-Feng Cai’s research was partially supported by Hong Kong RGC Grant GRF 16310620 and GRF 16309219. Dong Xia’s research was partially supported by Hong Kong RGC Grant ECS 26302019, GRF 16303320 and GRF 16300121.

Robust matrix factor analysis method with adaptive parameter adjustment using Cauchy weighting

Statistical Inference for Large-dimensional Matrix Factor Model from Least Squares and Huber Loss Points of View

Matrix Factor Analysis: from Least Squares to Iterative Projection

Robust Asymmetric Bayesian Adaptive Matrix Factorization

A NEW PERSPECTIVE ON ROBUST M-ESTIMATION: FINITE SAMPLE THEORY AND APPLICATIONS TO DEPENDENCE-ADJUSTED MULTIPLE TESTING

Estimating and Testing for Smooth Structural Changes in Moment Condition Models

Robust Factor Analysis Parameter Estimation

Adaptive Shrinkage Estimation of High Dimensional Moment Condition Model with Smooth Structural Changes

Bayesian factor-adjusted sparse regression

Huber Principal Component Analysis for Large-dimensional Factor Models

Robust matrix estimations meet Frank–Wolfe algorithm

Adaptive Robust Large Volatility Matrix Estimation Based on High-Frequency Financial Data

Robust Bayesian Model Averaging for Linear Regression Models With Heavy-Tailed Errors

Computationally Efficient and Statistically Optimal Robust Low-rank Matrix Estimation

Robust M-Estimation for Additive Single-Index Cointegrating Time Series Models

Regularized covariance matrix estimation in high dimensional approximate factor models

Adaptive Huber Regression on Markov-dependent Data

Robust Matrix Completion with Heavy-tailed Noise

Reweighted Quasi Norm Regularized Low-Rank Factorization for Matrix Robust PCA

Robust matrix completion via Novel M-estimator Functions

A Scalable Frequentist Model Averaging Method