Abstract:This paper focuses on convex constrained optimization problems, where the solution is subject to a convex inequality constraint. In particular, we aim at challenging problems for which both projection into the constrained domain and a linear optimization under the inequality constraint are time-consuming, which render both projected gradient methods and conditional gradient methods (a.k.a. the Frank-Wolfe algorithm) expensive. In this paper, we develop projection reduced optimization algorithms for both smooth and non-smooth optimization with improved convergence rates under a certain regularity condition of the constraint function. We first present a general theory of optimization with only one projection. Its application to smooth optimization with only one projection yields $O(1/\epsilon)$ iteration complexity, which improves over the $O(1/\epsilon^2)$ iteration complexity established before for non-smooth optimization and can be further reduced under strong convexity. Then we introduce a local error bound condition and develop faster algorithms for non-strongly convex optimization at the price of a logarithmic number of projections. In particular, we achieve an iteration complexity of $\widetilde O(1/\epsilon^{2(1-\theta)})$ for non-smooth optimization and $\widetilde O(1/\epsilon^{1-\theta})$ for smooth optimization, where $\theta\in(0,1]$ appearing the local error bound condition characterizes the functional local growth rate around the optimal solutions. Novel applications in solving the constrained $\ell_1$ minimization problem and a positive semi-definite constrained distance metric learning problem demonstrate that the proposed algorithms achieve significant speed-up compared with previous algorithms.

Refinements of the Norm of Two Orthogonal Projections

Inequalities for linear combinations of orthogonal projections and applications

Some Refinement of the Inequality in Quasi-2-normed Spaces and Quasi-(2;P)-normed Spaces

Products of two orthogonal projections

Norm Inequalities for Hilbert space operators with Applications

New extrapolation projection contraction algorithms based on the golden ratio for pseudo-monotone variational inequalities

A Theoretical Comparison of Two Classes of Projection Methods

Improved strategies of the Equality Set Projection (ESP) algorithm for computing polytope projection

Geometric Characterization of the Numerical Range of Parallel Sum of Two Orthogonal Projections

Composite orthogonal projection methods for large matrix eigenproblems

Projections in normed linear spaces and sufficient enlargements

Norm inequalities in and a geometric constant

Doubling the rate -- improved error bounds for orthogonal projection with application to interpolation

Projection and contraction method with double inertial steps for quasi-monotone variational inequalities

A Richer Theory of Convex Constrained Optimization with Reduced Projections and Improved Rates

Optimal linear approximation and isometric extensions

A class modified projection algorithms for nonmonotone variational inequalities with continuity

An improved relaxed inertial projection algorithm for solving the minimum-norm solution of variational inequality and fixed point problems

Covering Constants for Metric Projection Operator with Applications to Stochastic Fixed-Point Problems

Some Theoretical Comparisons of Refined Ritz Vectors and Ritz Vectors

An Exact Penalty Approach for Optimization with Nonnegative Orthogonality Constraints