Abstract:Low-rank matrix estimation is a canonical problem that finds numerous applications in signal processing, machine learning and imaging science. A popular approach in practice is to factorize the matrix into two compact low-rank factors, and then optimize these factors directly via simple iterative methods such as gradient descent and alternating minimization. Despite nonconvexity, recent literatures have shown that these simple heuristics in fact achieve linear convergence when initialized properly for a growing number of problems of interest. However, upon closer examination, existing approaches can still be computationally expensive especially for ill-conditioned matrices: the convergence rate of gradient descent depends linearly on the condition number of the low-rank matrix, while the per-iteration cost of alternating minimization is often prohibitive for large matrices. The goal of this paper is to set forth a competitive algorithmic approach dubbed Scaled Gradient Descent (ScaledGD) which can be viewed as pre-conditioned or diagonally-scaled gradient descent, where the pre-conditioners are adaptive and iteration-varying with a minimal computational overhead. With tailored variants for low-rank matrix sensing, robust principal component analysis and matrix completion, we theoretically show that ScaledGD achieves the best of both worlds: it converges linearly at a rate independent of the condition number of the low-rank matrix similar as alternating minimization, while maintaining the low per-iteration cost of gradient descent. Our analysis is also applicable to general loss functions that are restricted strongly convex and smooth over low-rank matrices. To the best of our knowledge, ScaledGD is the first algorithm that provably has such properties over a wide range of low-rank matrix estimation tasks.

Accelerated Gradient Method for A Class of Nonconvex Low Rank Problem: Essentially Matching the Optimal Convex Convergence Rate

Provable accelerated gradient method for nonconvex low rank optimization

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

Projected Gradient Descent Algorithm for Low-Rank Matrix Estimation

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees

Robust Recovery of Low-Rank Matrices Via Non-Convex Optimization

The Global Optimization Geometry of Low-Rank Matrix Optimization

Preconditioning Matters: Fast Global Convergence of Non-convex Matrix Factorization via Scaled Gradient Descent

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Fast Stochastic Algorithms for Low-rank and Nonsmooth Matrix Problems

Fast Low-Rank Matrix Learning with Nonconvex Regularization

Gauss-Southwell type descent methods for low-rank matrix optimization

Fast Gradient Method for Low-Rank Matrix Estimation

Exact Linear Convergence Rate Analysis for Low-Rank Symmetric Matrix Completion via Gradient Descent

Low-Rank Extragradient Methods for Scalable Semidefinite Optimization

Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview

Efficient Low-rank Identification via Accelerated Iteratively Reweighted Nuclear Norm Minimization

Gradient descent with nonconvex constraints: local concavity determines convergence

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems