Abstract:In this work, we study the performance of sub-gradient method (SubGM) on a natural nonconvex and nonsmooth formulation of low-rank matrix recovery with $\ell_1$-loss, where the goal is to recover a low-rank matrix from a limited number of measurements, a subset of which may be grossly corrupted with noise. We study a scenario where the rank of the true solution is unknown and over-estimated instead. The over-estimation of the rank gives rise to an over-parameterized model in which there are more degrees of freedom than needed. Such over-parameterization may lead to overfitting, or adversely affect the performance of the algorithm. We prove that a simple SubGM with small initialization is agnostic to both over-parameterization and noise in the measurements. In particular, we show that small initialization nullifies the effect of over-parameterization on the performance of SubGM, leading to an exponential improvement in its convergence rate. Moreover, we provide the first unifying framework for analyzing the behavior of SubGM under both outlier and Gaussian noise models, showing that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values, and--perhaps surprisingly--even if the globally optimal solutions do not correspond to the ground truth. At the core of our results is a robust variant of restricted isometry property, called Sign-RIP, which controls the deviation of the sub-differential of the $\ell_1$-loss from that of an ideal, expected loss. As a byproduct of our results, we consider a subclass of robust low-rank matrix recovery with Gaussian measurements, and show that the number of required samples to guarantee the global convergence of SubGM is independent of the over-parameterized rank.

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Convergence Analysis of Projected Gradient Descent for Schatten-p Nonconvex Matrix Recovery

One-Bit Compressive Sensing with Projected Subgradient Method under Sparsity Constraints

The Convergence Guarantees of a Non-Convex Approach for Sparse Recovery.

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

From Least Squares to Sparse: A Non-Convex Approach with Guarantee

A Non-convex Approach for Sparse Recovery with Convergence Guarantee

A Low-rank Projected Proximal Gradient Method for Spectral Compressed Sensing

A Unified Computational and Statistical Framework for Nonconvex Low-rank Matrix Estimation

Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization

Low-rank matrix recovery with non-quadratic loss: projected gradient method and regularity projection oracle

Constrained Optimization with Compressed Gradients: A Dynamical Systems Perspective

Fast Sparse Recovery Via Non-Convex Optimization

Projected Gradient Descent Algorithm for Low-Rank Matrix Estimation

A Unified Variance Reduction-Based Framework for Nonconvex Low-Rank Matrix Recovery

Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing

Subspace projection matrix recovery from incomplete information

An Alternating Manifold Proximal Gradient Method for Sparse Principal Component Analysis and Sparse Canonical Correlation Analysis

A Proximal-Gradient Homotopy Method for the Sparse Least-Squares Problem

Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees