Abstract:This paper rigorously shows how over-parameterization changes the convergence behaviors of gradient descent (GD) for the matrix sensing problem, where the goal is to recover an unknown low-rank ground-truth matrix from near-isotropic linear measurements. First, we consider the symmetric setting with the symmetric parameterization where $M^* \in \mathbb{R}^{n \times n}$ is a positive semi-definite unknown matrix of rank $r \ll n$, and one uses a symmetric parameterization $XX^\top$ to learn $M^*$. Here $X \in \mathbb{R}^{n \times k}$ with $k > r$ is the factor matrix. We give a novel $\Omega (1/T^2)$ lower bound of randomly initialized GD for the over-parameterized case ($k >r$) where $T$ is the number of iterations. This is in stark contrast to the exact-parameterization scenario ($k=r$) where the convergence rate is $\exp (-\Omega (T))$. Next, we study asymmetric setting where $M^* \in \mathbb{R}^{n_1 \times n_2}$ is the unknown matrix of rank $r \ll \min\{n_1,n_2\}$, and one uses an asymmetric parameterization $FG^\top$ to learn $M^*$ where $F \in \mathbb{R}^{n_1 \times k}$ and $G \in \mathbb{R}^{n_2 \times k}$. Building on prior work, we give a global exact convergence result of randomly initialized GD for the exact-parameterization case ($k=r$) with an $\exp (-\Omega(T))$ rate. Furthermore, we give the first global exact convergence result for the over-parameterization case ($k>r$) with an $\exp(-\Omega(\alpha^2 T))$ rate where $\alpha$ is the initialization scale. This linear convergence result in the over-parameterization case is especially significant because one can apply the asymmetric parameterization to the symmetric setting to speed up from $\Omega (1/T^2)$ to linear convergence. On the other hand, we propose a novel method that only modifies one step of GD and obtains a convergence rate independent of $\alpha$, recovering the rate in the exact-parameterization case.

Hyperparameter tuning via trajectory predictions: Stochastic prox-linear methods in matrix sensing

Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Data

Alternating minimization for generalized rank one matrix sensing: Sharp predictions from a random initialization

Active Probabilistic Inference on Matrices for Pre-Conditioning in Stochastic Optimization

The stochastic proximal distance algorithm

High-probability Convergence Bounds for Nonlinear Stochastic Gradient Descent Under Heavy-tailed Noise

Stochastic Newton Proximal Extragradient Method

Stochastic Primal-Dual Algorithms with Faster Convergence Than O(1/√T) for Problems Without Bilinear Structure

Sharper Bounds for Proximal Gradient Algorithms with Errors

Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure

Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent

Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Optimal and instance-dependent guarantees for Markovian linear stochastic approximation

Breaking the Heavy-Tailed Noise Barrier in Stochastic Optimization Problems

Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guarantees

How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization

Improving Linear System Solvers for Hyperparameter Optimisation in Iterative Gaussian Processes

Stochastic Proximal Gradient Algorithm with Minibatches. Application to Large Scale Learning Models

Modified Gauss-Newton Algorithms under Noise