Abstract:Adaptive gradient methods have shown excellent performances for solving many machine learning problems. Although multiple adaptive gradient methods were recently studied, they mainly focus on either empirical or theoretical aspects and also only work for specific problems by using some specific adaptive learning rates. Thus, it is desired to design a universal framework for practical algorithms of adaptive gradients with theoretical guarantee to solve general problems. To fill this gap, we propose a faster and universal framework of adaptive gradients (i.e., SUPER-ADAM) by introducing a universal adaptive matrix that includes most existing adaptive gradient forms. Moreover, our framework can flexibly integrate the momentum and variance reduced techniques. In particular, our novel framework provides the convergence analysis support for adaptive gradient methods under the nonconvex setting. In theoretical analysis, we prove that our SUPER-ADAM algorithm can achieve the best known gradient (i.e., stochastic first-order oracle (SFO)) complexity of $\tilde{O}(\epsilon^{-3})$ for finding an $\epsilon$-stationary point of nonconvex optimization, which matches the lower bound for stochastic smooth nonconvex optimization. In numerical experiments, we employ various deep learning tasks to validate that our algorithm consistently outperforms the existing adaptive algorithms. Code is available at <a class="link-external link-https" href="https://github.com/LIJUNYI95/SuperAdam" rel="external noopener nofollow">this https URL</a>

Adaptive first-order methods with enhanced worst-case rates

Gradient Methods with Memory for Minimizing Composite Functions

The Speed-Robustness Trade-Off for First-Order Methods with Additive Gradient Noise

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness

Nonasymptotic Analysis of Accelerated Methods With Inexact Oracle Under Absolute Error Bound

A New Class of Composite Objective Multi-step Estimating-sequence Techniques (COMET)

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients

Fast Multiobjective Gradient Methods with Nesterov Acceleration via Inertial Gradient-Like Systems

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

A Note on Nesterov's Accelerated Method in Nonconvex Optimization: a Weak Estimate Sequence Approach

Nonlinear conjugate gradient methods: worst-case convergence rates via computer-assisted analyses

OPTAMI: Global Superlinear Convergence of High-order Methods

A template for gradient norm minimization

A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems

Intermediate Gradient Methods with Relative Inexactness

Heuristic adaptive fast gradient method in stochastic optimization tasks

Accelerated First-Order Optimization under Nonlinear Constraints

Accelerated Algorithms for Constrained Nonconvex-Nonconcave Min-Max Optimization and Comonotone Inclusion

Adaptive generalized conditional gradient method for multiobjective optimization