Abstract:Recently, there has been significant progress in understanding the convergence and generalization properties of gradient-based methods for training overparameterized learning models. However, many aspects including the role of small random initialization and how the various parameters of the model are coupled during gradient-based updates to facilitate good generalization remain largely mysterious. A series of recent papers have begun to study this role for non-convex formulations of symmetric Positive Semi-Definite (PSD) matrix sensing problems which involve reconstructing a low-rank PSD matrix from a few linear measurements. The underlying symmetry/PSDness is crucial to existing convergence and generalization guarantees for this problem. In this paper, we study a general overparameterized low-rank matrix sensing problem where one wishes to reconstruct an asymmetric rectangular low-rank matrix from a few linear measurements. We prove that an overparameterized model trained via factorized gradient descent converges to the low-rank matrix generating the measurements. We show that in this setting, factorized gradient descent enjoys two implicit properties: (1) coupling of the trajectory of gradient descent where the factors are coupled in various ways throughout the gradient update trajectory and (2) an algorithmic regularization property where the iterates show a propensity towards low-rank models despite the overparameterized nature of the factorized model. These two implicit properties in turn allow us to show that the gradient descent trajectory from small random initialization moves towards solutions that are both globally optimal and generalize well.

Balancing Principle in Supervised Learning for a General Regularization Scheme

Lepskii Principle in Supervised Learning

Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions

Generalization errors of Laplacian regularized least squares regression

A New Principle for Tuning-Free Huber Regression

Multi-parameter Regularization and Its Numerical Realization

On the Balancing Principle for Some Problems of Numerical Analysis.

A Statistical Theory of Regularization-Based Continual Learning

A Duality Approach to Regularized Learning Problems in Banach Spaces

On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning

Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent

Optimal Regularization Parameter Estimation for Spectral Regression Discriminant Analysis

Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing

Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression

Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced

Adversarial confidence and smoothness regularizations for scalable unsupervised discriminative learning

Random Smoothing Regularization in Kernel Gradient Descent Learning

Bregman Divergence-Based Regularization for Transfer Subspace Learning

Discrepancy Curves for Multi-Parameter Regularization

Parameter-Agnostic Optimization under Relaxed Smoothness

Making SGD Parameter-Free