Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

On Difference‐based Gradient Estimation in Nonparametric Regression

Pathwise Derivatives for Multivariate Distributions

Nonparametric Estimation via Partial Derivatives

Stochastic Gradient Descent for Nonparametric Regression

Difference-based covariance matrix estimate in time series nonparametric regression with applications to specification tests

Difference-based covariance matrix estimation in time series nonparametric regression with application to specification tests

Eliminating Ratio Bias for Gradient-based Simulated Parameter Estimation

An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations

Difference-based estimation for error variances in repeated measurement regression models

Variance-Reduced Gradient Estimator for Nonconvex Zeroth-Order Distributed Optimization

Linear convergence of decentralized estimation for statistical estimation using gradient method

A Spectral Approach to Gradient Estimation for Implicit Distributions

Gradient Estimation with Discrete Stein Operators

Parametric estimation of stochastic differential equations via online gradient descent

A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

A nonparametric regression alternative to empirical Bayes approaches to simultaneous estimation

Simultaneous Estimation and Dataset Selection for Transfer Learning in High Dimensions by a Non-convex Penalty

ACF estimation via difference schemes for a semiparametric model with m-dependent errors

Non-linear Gradient Algorithm for Parameter Estimation: Extended version