Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

Graduated Optimization of Black-Box Functions

Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning

Adaptive Local Bayesian Optimization Over Multiple Discrete Variables

When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario

Towards Differentiable Multilevel Optimization: A Gradient-Based Approach

High-dimensional Bayesian optimization using low-dimensional feature spaces

Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization with Optimal Noise Scheduling

Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm

Derivative-Free Optimization with Adaptive Experience for Efficient Hyper-Parameter Tuning.

Gradient Descent: The Ultimate Optimizer

A Gradient-based Bilevel Optimization Approach for Tuning Hyperparameters in Machine Learning

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Random Function Descent

Scalable Nested Optimization for Deep Learning

Bayesian Optimisation for Constrained Problems

Learning to optimize with convergence guarantees using nonlinear system theory

An adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning

Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization

Black-box Optimization Algorithms for Regularized Least-squares Problems

Tuning-Free Bilevel Optimization: New Algorithms and Convergence Analysis

Generative Evolutionary Strategy For Black-Box Optimizations