Abstract:Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order to guarantee fast convergence. For vanilla procedures such as gradient descent, however, prior theory either recommends highly conservative learning rates to avoid overshooting, or completely lacks performance guarantees. This paper uncovers a striking phenomenon in nonconvex optimization: even in the absence of explicit regularization, gradient descent enforces proper regularization implicitly under various statistical models. In fact, gradient descent follows a trajectory staying within a basin that enjoys nice geometry, consisting of points incoherent with the sampling mechanism. This "implicit regularization" feature allows gradient descent to proceed in a far more aggressive fashion without overshooting, which in turn results in substantial computational savings. Focusing on three fundamental statistical estimation problems, i.e. phase retrieval, low-rank matrix completion, and blind deconvolution, we establish that gradient descent achieves near-optimal statistical and computational guarantees without explicit regularization. In particular, by marrying statistical modeling with generic optimization theory, we develop a general recipe for analyzing the trajectories of iterative algorithms via a leave-one-out perturbation argument. As a byproduct, for noisy matrix completion, we demonstrate that gradient descent achieves near-optimal error control --- measured entrywise and by the spectral norm --- which might be of independent interest.

Regularization of Riemannian optimization: Application to process tomography and quantum machine learning

Fast Quantum Process Tomography via Riemannian Gradient Descent

Randomized Gradient Descents on Riemannian Manifolds: Almost Sure Convergence to Global Minima in and beyond Quantum Optimization

Gradient-descent quantum process tomography by learning Kraus operators

On the regularization and optimization in quantum detector tomography

Using non-convex optimization in quantum process tomography: Factored gradient descent is tough to beat

Improving Gradient Methods via Coordinate Transformations: Applications to Quantum Machine Learning

Optimizing quantum circuits with Riemannian gradient flow

Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution

Reconstruction of Quantum Channel Via Convex Optimization.

Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms

Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisation

Continuous-variable quantum Gaussian process regression and quantum singular value decomposition of non-sparse low rank matrices

Adaptive Quadratically Regularized Newton Method for Riemannian Optimization

Quantum Natural Stochastic Pairwise Coordinate Descent

Convergence rates for critical point regularization

Regularization of Limited Memory Quasi-Newton Methods for Large-Scale Nonconvex Minimization

Improving Trainability of Variational Quantum Circuits via Regularization Strategies

Riemannian optimization of isometric tensor networks

Smoothing $\mathcal{L}^2$ gradients in iterative regularization