Abstract:Low-rank matrix estimation is a canonical problem that finds numerous applications in signal processing, machine learning and imaging science. A popular approach in practice is to factorize the matrix into two compact low-rank factors, and then optimize these factors directly via simple iterative methods such as gradient descent and alternating minimization. Despite nonconvexity, recent literatures have shown that these simple heuristics in fact achieve linear convergence when initialized properly for a growing number of problems of interest. However, upon closer examination, existing approaches can still be computationally expensive especially for ill-conditioned matrices: the convergence rate of gradient descent depends linearly on the condition number of the low-rank matrix, while the per-iteration cost of alternating minimization is often prohibitive for large matrices. The goal of this paper is to set forth a competitive algorithmic approach dubbed Scaled Gradient Descent (ScaledGD) which can be viewed as pre-conditioned or diagonally-scaled gradient descent, where the pre-conditioners are adaptive and iteration-varying with a minimal computational overhead. With tailored variants for low-rank matrix sensing, robust principal component analysis and matrix completion, we theoretically show that ScaledGD achieves the best of both worlds: it converges linearly at a rate independent of the condition number of the low-rank matrix similar as alternating minimization, while maintaining the low per-iteration cost of gradient descent. Our analysis is also applicable to general loss functions that are restricted strongly convex and smooth over low-rank matrices. To the best of our knowledge, ScaledGD is the first algorithm that provably has such properties over a wide range of low-rank matrix estimation tasks.

Greedy Bilateral Sketch, Completion & Smoothing.

Greedy Learning of Generalized Low-Rank Models.

Learning of Generalized Low-Rank Models: A Greedy Approach.

Nearly-optimal Robust Matrix Completion

Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares

Seeing the Forest from the Trees in Two Looks: Matrix Sketching by Cascaded Bilateral Sampling

Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices Via Convex Optimization.

Generalized Least Squares Kernelized Tensor Factorization

Scalable Robust Matrix Recovery: Frank-Wolfe Meets Proximal Methods

GoDec: Randomized Low-Rank & Sparse Matrix Decomposition in Noisy Case

Adaptive Rank-One Matrix Completion using Sum of Outer Products

Learning-Based Low-Rank Approximations

Generalized Low-Rank Matrix Completion Model with Overlapping Group Error Representation

Robust Matrix Completion Based on Factorization and Truncated-Quadratic Loss Function

Accelerated Low-Rank Visual Recovery by Random Projection

Robust Recovery Via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization

Robust Matrix Completion with Deterministic Sampling via Convex Optimization

Outlier-robust sparse/low-rank least-squares regression and robust matrix completion

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

High Dimensional Robust Sparse Regression

Completing Low-Rank Matrices with Corrupted Samples from Few Coefficients in General Basis