Abstract:This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render both projected gradient methods and conditional gradient methods (a.k.a. the Frank-Wolfe algorithm) expensive. In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function. We first present a general theory of optimization with only one projection. Its application to smooth optimization with only one projection yields $O(1/\epsilon)$ iteration complexity, which improves over the $O(1/\epsilon^2)$ iteration complexity established before for non-smooth optimization and can be further reduced under strong convexity. Then we introduce a local error bound condition and develop faster algorithms for non-strongly convex optimization at the price of a logarithmic number of projections. In particular, we achieve an iteration complexity of $\widetilde O(1/\epsilon^{2(1-\theta)})$ for non-smooth optimization and $\widetilde O(1/\epsilon^{1-\theta})$ for smooth optimization, where $\theta\in(0,1]$ appearing the local error bound condition characterizes the functional local growth rate around the optimal solutions. Novel applications in solving the constrained $\ell_1$ minimization problem and a positive semi-definite constrained distance metric learning problem demonstrate that the proposed algorithms achieve significant speed-up compared with previous algorithms.

Projected shrinkage algorithm for box-constrained ℓ _1 -minimization

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

An accelerated proximal iterative hard thresholding method for $\ell_0$ minimization

A New Proximal Iterative Hard Thresholding Method with Extrapolation for $$\ell _0$$ Minimization

Efficient Projected Gradient Methods for L 0 Constrained Optimization

ProPPA: A Fast Algorithm for $\ell_1$ Minimization and Low-Rank Matrix Completion

Solving the L1 regularized least square problem via a box-constrained smooth minimization

The variant of the iterative shrinkage-thresholding algorithm for minimization of the l 1 over l ∞ norms

Proximal Projection Method for Stable Linearly Constrained Optimization

The Variant of the Iterative Shrinkage-Thresholding Algorithm for Minimization of the ℓ1 over ℓ∞ Norms

An Inertial Shrinking Projection Algorithm for Split Common Fixed Point Problems

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

A family of multi-parameterized proximal point algorithms

An interior proximal gradient method for nonconvex optimization

A projected-search interior-point method for nonlinearly constrained optimization

On the Convergence of an Active-Set Method for ℓ1 Minimization

Inertial hybrid and shrinking projection algorithms for solving variational inequality problems

A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates

An improved relaxed inertial projection algorithm for solving the minimum-norm solution of variational inequality and fixed point problems