Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

A Memory Gradient Method Based on the Nonmonotone Technique

The convergence of conjugate gradient method with nonmonotone line search

Memory gradient method for multiobjective optimization

Towards Memory-Friendly Deterministic Incremental Gradient Method.

A derivative-free memoryless BFGS hyperplane projection method for solving large-scale nonlinear monotone equations

Gradient Methods with Memory for Minimizing Composite Functions

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

Continuation Newton methods with deflation techniques for global optimization problems

Limited memory gradient methods for unconstrained optimization

A new non-adaptive optimization method: Stochastic gradient descent with momentum and difference

An Adaptive Stochastic Gradient Method with Non-negative Gauss-Newton Stepsizes

A self-scaling memoryless BFGS based conjugate gradient method using multi-step secant condition for unconstrained minimization

A gradient descent akin method for constrained optimization: algorithms and applications

Towards Differentiable Multilevel Optimization: A Gradient-Based Approach

A new hybrid conjugate gradient method close to the memoryless BFGS quasi-Newton method and its application in image restoration and machine learning

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

A nonmonotone trust region method for unconstrained optimization

A family of limited memory three term conjugate gradient methods

On Markov Chain Gradient Descent