Abstract:This paper presents new first-order methods for achieving optimal oracle complexities in convex optimization with convex functional constraints. Oracle complexities are measured by the number of function and gradient evaluations. To achieve this, we enable first-order methods to utilize computational oracles for solving diagonal quadratic programs in subproblems. For problems where the optimal value $f^*$ is known, such as those in overparameterized models and feasibility problems, we propose an accelerated first-order method that incorporates a modified Polyak step size and Nesterov's momentum. Notably, our method does not require knowledge of smoothness levels, Hölder continuity parameter of the gradient, or additional line search, yet achieves the optimal oracle complexity bound of $\mathcal{O}(\varepsilon^{-2/(1+3\rho)})$ under Hölder smoothness conditions. When $f^*$ is unknown, we reformulate the problem as finding the root of the optimal value function and develop inexact fixed-point iteration and secant method to compute $f^*$. These root-finding subproblems are solved inexactly using first-order methods to a specified relative accuracy. We employ the accelerated prox-level (APL) method, which is proven to be uniformly optimal for convex optimization with simple constraints. Our analysis demonstrates that APL-based level-set methods also achieve the optimal oracle complexity of $\mathcal{O}(\varepsilon^{-2/(1+3\rho)})$ for convex function-constrained optimization, without requiring knowledge of any problem-specific structures. Through experiments on various tasks, we demonstrate the advantages of our methods over existing approaches in function-constrained optimization.

Fast First-Order Methods for Minimizing Convex Composite Functions

Adaptive Accelerated Composite Minimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Adaptive Proximal Average Approximation for Composite Convex Minimization.

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

Uniformly Optimal and Parameter-free First-order Methods for Convex and Function-constrained Optimization

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Parameter-Free Accelerated Gradient Descent for Nonconvex Minimization

Accelerated Bregman gradient methods for relatively smooth and relatively Lipschitz continuous minimization problems

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Zeroth-Order Katyusha: An Accelerated Derivative-Free Method for Composite Convex Optimization

Fast Stochastic Composite Minimization and an Accelerated Frank-Wolfe Algorithm under Parallelization

First-order Methods for Affinely Constrained Composite Non-convex Non-smooth Problems: Lower Complexity Bound and Near-optimal Methods

Complexity-optimal and parameter-free first-order methods for finding stationary points of composite optimization problems

On Optimal Universal First-Order Methods for Minimizing Heterogeneous Sums

An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization

An Average Curvature Accelerated Composite Gradient Method for Nonconvex Smooth Composite Optimization Problems

Adaptive first-order methods with enhanced worst-case rates

Minimization Over the Nonconvex Sparsity Constraint Using A Hybrid First-order method

A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems