Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Regularization independent of the noise level: an analysis of quasi-optimality

Small noise analysis for Tikhonov and RKHS regularizations

On the Asymptotical Regularization for Linear Inverse Problems in Presence of White Noise

Sparse Estimation Via ℓ_q Optimization Method in High-Dimensional Linear Regression

Regularization of linear ill-posed problems involving multiplication operators

Quasi-maximum likelihood estimation and penalized estimation under non-standard conditions

Generalized Qualification and Qualification Levels for Spectral Regularization Methods

Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms

Regularization of Discrete Ill-Conditioned Problems Done Right -- I

Quantitative Convergence of Quadratically Regularized Linear Programs

On Learning the Optimal Regularization Parameter in Inverse Problems

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Off-the-grid regularisation for Poisson inverse problems

On the Generalized Discrepancy Principle for Tikhonov Regularization in Hilbert Scales

Constrained Optimization in the Presence of Noise

Near-Optimal Estimation of Linear Functionals with Log-Concave Observation Errors

On the order optimality of the regularization via inexact Newton iterations

Monotonicity in Quadratically Regularized Linear Programs

Superiorized Regularization of Inverse Problems

Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise

A Relationship Between the False Alarm Probability and the Atomic Norm Denoising Regularization Parameter for Line Spectral Estimation