Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Semiparametric Model And Superefficiency In Blind Deconvolution

Multi-frame Blind Deconvolution Using Sparse Priors

Lp (1≤p≤∞) BLIND ADAPTIVE DECONVOLUTION AND ITS APPLICATIONS

Blind Deconvolution Using an Improved L0 Sparse Representation

L∞ Blind Deconvolution for the Generalized AR Model.

Semiparametric Approach to Multichannel Blind Deconvolution of Nonminimum Phase Systems

Identifiability of Multichannel Blind Deconvolution and Nonconvex Regularization Algorithm

Non-blind image deconvolution using natural image gradient prior

Marginal Likelihood Estimation in Semiblind Image Deconvolution: A Stochastic Approximation Approach

Adaptive Multichannel Blind Deconvolution Using State-Space Models

A Novel Approach Based on Differential Evolution for Blind Deconvolution.

Rapid, Robust, and Reliable Blind Deconvolution Via Nonconvex Optimization

Gradient-based discriminative modeling for blind image deblurring

Generalized variational framework with minimax optimization for parametric blind deconvolution

Kalman Filter and State-Space Approach to Blind Deconvolution

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

A stochastic optimisation unadjusted Langevin method for empirical Bayesian estimation in semi-blind image deblurring problems

An Efficient Blind Image Deblurring Algorithm

Blind Image Deblurring Using Spectral Properties of Convolution Operators

An Investigation on Semi-Blind Deconvolution by Using Nonlinear-Programming

Efficient non-blind deconvolution method for large scale blurred image with hybrid regularizations