Abstract:This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render both projected gradient methods and conditional gradient methods (a.k.a. the Frank-Wolfe algorithm) expensive. In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function. We first present a general theory of optimization with only one projection. Its application to smooth optimization with only one projection yields $O(1/\epsilon)$ iteration complexity, which improves over the $O(1/\epsilon^2)$ iteration complexity established before for non-smooth optimization and can be further reduced under strong convexity. Then we introduce a local error bound condition and develop faster algorithms for non-strongly convex optimization at the price of a logarithmic number of projections. In particular, we achieve an iteration complexity of $\widetilde O(1/\epsilon^{2(1-\theta)})$ for non-smooth optimization and $\widetilde O(1/\epsilon^{1-\theta})$ for smooth optimization, where $\theta\in(0,1]$ appearing the local error bound condition characterizes the functional local growth rate around the optimal solutions. Novel applications in solving the constrained $\ell_1$ minimization problem and a positive semi-definite constrained distance metric learning problem demonstrate that the proposed algorithms achieve significant speed-up compared with previous algorithms.

Performance Estimation for Smooth and Strongly Convex Sets

Gauges and Accelerated Optimization over Smooth and/or Strongly Convex Sets

Interpolation Conditions for Linear Operators and Applications to Performance Estimation Problems

A constraint-based approach to function interpolation, with application to performance estimation for weakly convex optimisation

Non-Euclidean High-Order Smooth Convex Optimization

Convex Discrete Optimization

Scalable Projection-Free Optimization Methods via MultiRadial Duality Theory

On the Oracle Complexity of Higher-Order Smooth Non-Convex Finite-Sum Optimization

Approximating optimal solutions to biconvex parametric programs

A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles

Uniformly Optimal and Parameter-free First-order Methods for Convex and Function-constrained Optimization

Automated Lyapunov Analysis of Primal-Dual Optimization Algorithms: An Interpolation Approach

Linearization Algorithms for Fully Composite Optimization

An Inexact Primal-Dual Smoothing Framework for Large-Scale Non-Bilinear Saddle Point Problems

A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates

High-order Moreau envelope beyond convexity: An inexact two-level smoothing framework

Principled analyses and design of first-order methods with inexact proximal operators

Accelerated First-Order Optimization under Nonlinear Constraints

Optimization Algorithms for Faster Computational Geometry.

Analysis of Optimization Algorithms via Sum-of-Squares

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints