Abstract:This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render both projected gradient methods and conditional gradient methods (a.k.a. the Frank-Wolfe algorithm) expensive. In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function. We first present a general theory of optimization with only one projection. Its application to smooth optimization with only one projection yields $O(1/\epsilon)$ iteration complexity, which improves over the $O(1/\epsilon^2)$ iteration complexity established before for non-smooth optimization and can be further reduced under strong convexity. Then we introduce a local error bound condition and develop faster algorithms for non-strongly convex optimization at the price of a logarithmic number of projections. In particular, we achieve an iteration complexity of $\widetilde O(1/\epsilon^{2(1-\theta)})$ for non-smooth optimization and $\widetilde O(1/\epsilon^{1-\theta})$ for smooth optimization, where $\theta\in(0,1]$ appearing the local error bound condition characterizes the functional local growth rate around the optimal solutions. Novel applications in solving the constrained $\ell_1$ minimization problem and a positive semi-definite constrained distance metric learning problem demonstrate that the proposed algorithms achieve significant speed-up compared with previous algorithms.

Restricted strong convexity and its applications to convergence analysis of gradient-type methods in convex optimization

Convergence analysis of the Gauss–Newton method for convex inclusion and convex-composite optimization problems

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

Linear convergence of first order methods for non-strongly convex optimization

Convex and Non-convex Optimization Under Generalized Smoothness

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Convergence of Inexact Steepest Descent Algorithm for Multiobjective Optimizations on Riemannian Manifolds Without Curvature Constraints

Convergence rates analysis of Interior Bregman Gradient Method for Vector Optimization Problems

On Convergence of the Gauss-Newton Method for Convex Composite Optimization

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Error Bounds for Non-Polyhedral Convex Optimization and Applications to Linear Convergence of FDM and PGM.

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

An Extended Gradient Method for Smooth and Strongly Convex Functions

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates

On Convergence Rate of Distributed Stochastic Gradient Algorithm for Convex Optimization with Inequality Constraints.

Aug-PDG: Linear Convergence of Convex Optimization with Inequality Constraints

Aug-PDG: Linear Convergence of Convex Optimization with Inequality Constraints

On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

TWO FUNDAMENTAL CONVERGENCE THEOREMS FOR NONLINEAR CONJUGATE GRADIENT METHODS AND THEIR APPLICATIONS