Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Matrix Forms of Gradient Descent Algorithms Applied to Restoration of Blurred Images

Matrix Forms of Iterative Algorithms to Solve Large-Scale Discrete Ill-Posed Problems with an Application to Image Restoration.

Faster Gradient Descent and the Efficient Recovery of Images

Discrete Total Variation Model with Gradient Fidelity Term for Image Restoration

Blind Deconvolution Using an Improved L0 Sparse Representation

Solving a system of complex matrix equations using a gradient-based method and its application in image restoration

Image deblurring with matrix regression and gradient evolution

Generalized Proximal Conjugate Gradient Method for Image Deblurring

A Descent Conjugate Gradient Algorithm for Optimization Problems and Its Applications in Image Restoration and Compression Sensing

On image restoration problems using new conjugate gradient methods

Laplacian Smoothing Gradient Descent

Double inertial steps extragadient-type methods for solving optimal control and image restoration problems

Two Soft-Thresholding Based Iterative Algorithms for Image Deblurring

Gradient-based discriminative modeling for blind image deblurring

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

An Optimal Control Approach to Early Stopping Variational Methods for Image Restoration

Learning Deep Gradient Descent Optimization for Image Deconvolution

A Blind Image Deblurring Algorithm Based on Relative Gradient and Sparse Representation

Regularization Preconditioners for Frame-Based Image Deblurring with Reduced Boundary Artifacts

A class of new three-term descent conjugate gradient algorithms for large-scale unconstrained optimization and applications to image restoration problems

Motion Deblurring from a Single Image Using Gradient Enhancement.