Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Iterative regularization for low complexity regularizers

Smoothing $\mathcal{L}^2$ gradients in iterative regularization

A sparse optimization approach to infinite infimal convolution regularization

A fast iterative regularization method for ill-posed problems

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

An iteration regularization method with general convex penalty for nonlinear inverse problems in Banach spaces

A novel iterative integration regularization method for ill-posed inverse problems

A relaxed iterated Tikhonov regularization for linear ill-posed inverse problems

Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation

Superiorized Regularization of Inverse Problems

On regularization methods based on dynamic programming techniques

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

On Learning the Optimal Regularization Parameter in Inverse Problems

Reduced-Space Iteratively Reweighted Second-Order Methods for Nonconvex Sparse Regularization

An asymptotical regularization with convex constraints for inverse problems

Iterative regularization in classification via hinge loss diagonal descent

Smooth over-parameterized solvers for non-smooth structured optimization

Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms

Avoiding strict saddle points of nonconvex regularized problems

A data-driven iteratively regularized Landweber iteration

A nested primal–dual iterated Tikhonov method for regularized convex optimization