Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

On the Convergence of A Data-Driven Regularized Stochastic Gradient Descent for Nonlinear Ill-Posed Problems

On the Regularizing Property of Stochastic Gradient Descent

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Stochastic Gradient Descent for Linear Inverse Problems in Hilbert Spaces.

A Hybrid Steepest Descent Method For L-Infinity Geometry Problems

Stochastic Gradient Descent Method with Convex Penalty for Ill-Posed Problems in Banach Spaces

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Stochastic gradient descent for linear inverse problems in variable exponent Lebesgue spaces

A Fast Data-Driven Iteratively Regularized Method with Convex Penalty for Solving Ill-Posed Problems

Stochastic Data-Driven Bouligand Landweber Method for Solving Non-smooth Inverse Problems

Stochastic Gradient Descent in the Viewpoint of Graduated Optimization

A Descent Nonlinear Conjugate Gradient Method for Large-Scale Unconstrained Optimization.

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Gradient-Based Non-Linear Inverse Learning

A New Descent Nonlinear Conjugate Gradient Method for Unconstrained Optimization

On Linear Convergence of Adaptive Sign-based Gradient Descent

A Spectral Conjugate Gradient Method for Nonlinear Inverse Problems

On the Convergence and Improvement of Stochastic Normalized Gradient Descent

Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions

Stochastic Gradient Descent Revisited

Gradient Descent Methods for Regularized Optimization