Abstract:Gradient-based minimax optimal algorithms have greatly promoted the development of continuous optimization and machine learning. One seminal work due to Yurii Nesterov [Nes83a] established $\tilde{\mathcal{O}}(\sqrt{L/\mu})$ gradient complexity for minimizing an $L$-smooth $\mu$-strongly convex objective. However, an ideal algorithm would adapt to the explicit complexity of a particular objective function and incur faster rates for simpler problems, triggering our reconsideration of two defeats of existing optimization modeling and analysis. (i) The worst-case optimality is neither the instance optimality nor such one in reality. (ii) Traditional $L$-smoothness condition may not be the primary abstraction/characterization for modern practical problems. In this paper, we open up a new way to design and analyze gradient-based algorithms with direct applications in machine learning, including linear regression and beyond. We introduce two factors $(\alpha, \tau_{\alpha})$ to refine the description of the degenerated condition of the optimization problems based on the observation that the singular values of Hessian often drop sharply. We design adaptive algorithms that solve simpler problems without pre-known knowledge with reduced gradient or analogous oracle accesses. The algorithms also improve the state-of-art complexities for several problems in machine learning, thereby solving the open problem of how to design faster algorithms in light of the known complexity lower bounds. Specially, with the $\mathcal{O}(1)$-nuclear norm bounded, we achieve an optimal $\tilde{\mathcal{O}}(\mu^{-1/3})$ (v.s. $\tilde{\mathcal{O}}(\mu^{-1/2})$) gradient complexity for linear regression. We hope this work could invoke the rethinking for understanding the difficulty of modern problems in optimization.

An Aggressive Reduction on the Complexity of Optimization for Non-Strongly Convex Objectives

Improved Algorithms for Convex-Concave Minimax Optimization

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Minimization Over the Nonconvex Sparsity Constraint Using A Hybrid First-order method

Accelerated Gradient Algorithms with Adaptive Subspace Search for Instance-Faster Optimization

Optimality-based domain reduction for inequality-constrained NLP and MINLP problems

Optimal and parameter-free gradient minimization methods for convex and nonconvex optimization

Efficient Hyper-parameter Optimization with Cubic Regularization.

Parameter-Agnostic Optimization under Relaxed Smoothness

Obtaining Lower Query Complexities Through Lightweight Zeroth-Order Proximal Gradient Algorithms

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

A Proximal-Gradient Method for Constrained Optimization

Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient

Generalized-Smooth Nonconvex Optimization is As Efficient As Smooth Nonconvex Optimization

Regularized online exponentially concave optimization

Black-box Optimization Algorithms for Regularized Least-squares Problems

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks

A modified inertial proximal minimization algorithm for structured nonconvex and nonsmooth problem

Incremental Methods for Weakly Convex Optimization