Abstract:Low-rank matrix estimation is a canonical problem that finds numerous applications in signal processing, machine learning and imaging science. A popular approach in practice is to factorize the matrix into two compact low-rank factors, and then optimize these factors directly via simple iterative methods such as gradient descent and alternating minimization. Despite nonconvexity, recent literatures have shown that these simple heuristics in fact achieve linear convergence when initialized properly for a growing number of problems of interest. However, upon closer examination, existing approaches can still be computationally expensive especially for ill-conditioned matrices: the convergence rate of gradient descent depends linearly on the condition number of the low-rank matrix, while the per-iteration cost of alternating minimization is often prohibitive for large matrices. The goal of this paper is to set forth a competitive algorithmic approach dubbed Scaled Gradient Descent (ScaledGD) which can be viewed as pre-conditioned or diagonally-scaled gradient descent, where the pre-conditioners are adaptive and iteration-varying with a minimal computational overhead. With tailored variants for low-rank matrix sensing, robust principal component analysis and matrix completion, we theoretically show that ScaledGD achieves the best of both worlds: it converges linearly at a rate independent of the condition number of the low-rank matrix similar as alternating minimization, while maintaining the low per-iteration cost of gradient descent. Our analysis is also applicable to general loss functions that are restricted strongly convex and smooth over low-rank matrices. To the best of our knowledge, ScaledGD is the first algorithm that provably has such properties over a wide range of low-rank matrix estimation tasks.

A Self-Calibrated Direct Approach to Precision Matrix Estimation and Linear Discriminant Analysis in High Dimensions

Regularized Linear Discriminant Analysis Using a Nonlinear Covariance Matrix Estimator

Ultrahigh dimensional precision matrix estimation via refitted cross validation

Doubly Robust Semiparametric Inference Using Regularized Calibrated Estimation with High-dimensional Data

A new approach for ultrahigh dimensional precision matrix estimation

Inference for High-Dimensional Linear Expectile Regression with De-Biasing Method

A Linear Approach for Depth and Colour Camera Calibration Using Hybrid Parameters

Improved kernel direct Fisher discriminant analysis and face recognition

Precise Error Analysis of Regularized M-estimators in High-dimensions

Pivotal Estimation of Linear Discriminant Analysis in High Dimensions

Data-driven calibration of linear estimators with minimal penalties

A Consistent and Differentiable Lp Canonical Calibration Error Estimator

Robust high-dimensional precision matrix estimation

Estimation of Large Precision Matrices Through Block Penalization

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

High-Dimensional Linear Regression via Implicit Regularization

High-Dimensional Regularized Discriminant Analysis

A Method for Solving Ill-Conditioned Nonlinear Least Squares Problems and Its Application in Image Distortion Correction Using Self-Calibration

Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent

Two Stage Non-penalized Corrected Least Squares for High Dimensional Linear Models with Measurement error or Missing Covariates