Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

Accelerated Sparse Recovery Via Gradient Descent with Nonlinear Conjugate Gradient Momentum

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Multi-stage stochastic gradient method with momentum acceleration

A Globally Convergent Gradient Method with Momentum

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Accelerated Incremental Gradient Descent Using Momentum Acceleration with Scaling Factor.

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Robust Sparse Recovery Via Non-Convex Optimization

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

An accelerated conjugate gradient method with adaptive two-parameter with applications in image restoration

A Non-convex Approach for Sparse Recovery with Convergence Guarantee

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

The Convergence Guarantees of a Non-Convex Approach for Sparse Recovery.

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes