Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

Decreasing Gradient Optimization Algorithm with Variable Step Length Based on Chaotic Variables

A Kind Of Optimization Algorithm Based On Chaos

Antlion optimizer algorithm based on chaos search and its application

Gradient Methods with Adaptive Step-Sizes

An Improved Dynamical Evolutionary Algorithm Based on Chaotic

Two efficient gradient methods with approximately optimal stepsizes based on regularization models for unconstrained optimization

On the Accelerated Extremum Seeking Driven by Optimal Stochastic Perturbations

The Improved Stochastic Fractional Order Gradient Descent Algorithm

Barzilai-Borwein Step Size for Stochastic Gradient Descent

An Adaptive Shrinking Grid Search Chaotic Wolf Optimization Algorithm Using Standard Deviation Updating Amount

Optimization Calculation Method and Mathematical Modeling of Big Data Chaotic Model Based on Improved Genetic Algorithm

Shuffling-type gradient method with bandwidth-based step sizes for finite-sum optimization

ABNGrad: adaptive step size gradient descent for optimizing neural networks

Chaotic gradient based optimizer for solving multidimensional unconstrained and constrained optimization problems

An Algebraically Converging Stochastic Gradient Descent Algorithm for Global Optimization

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

A New Simple Stochastic Gradient Descent Type Algorithm With Lower Computational Complexity for Bilevel Optimization

A Gradient Smoothed Functional Algorithm with Truncated Cauchy Random Perturbations for Stochastic Optimization

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Improved teaching–learning-based optimization algorithm with Cauchy mutation and chaotic operators

Stochastic Multiple Chaotic Local Search-Incorporated Gradient-Based Optimizer