Convergence of weighted averages of relaxed projections

Ryszard Szwarc

DOI: https://doi.org/10.48550/arXiv.0709.2245

2007-09-18

Abstract:The convergence of the algorithm for solving convex feasibility problem is studied by the method of sequential averaged and relaxed projections. Some results of H. H. Bauschke and J. M. Borwein are generalized by introducing new methods. Examples illustrating these generalizations are given.

Metric Geometry

What problem does this paper attempt to address?

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Hang Xu,Song Li,Junhong Lin

DOI: https://doi.org/10.1007/s10444-024-10163-2

2024-01-01

Advances in Computational Mathematics

Abstract:Many problems in data science can be treated as recovering structural signals from a set of linear measurements, sometimes perturbed by dense noise or sparse corruptions. In this paper, we develop a unified framework of considering a nonsmooth formulation with sparse or low-rank constraint for meeting the challenges of mixed noises—bounded noise and sparse noise. We show that the nonsmooth formulations of the problems can be well solved by the projected subgradient methods at a rapid rate when initialized at any points. Consequently, nonsmooth loss functions ( ℓ _1 -minimization programs) are naturally robust against sparse noise. Our framework simplifies and generalizes the existing analyses including compressed sensing, matrix sensing, quadratic sensing, and bilinear sensing. Motivated by recent work on the stochastic gradient method, we also give some experimentally and theoretically preliminary results about the projected stochastic subgradient method.
Convergence Rate Analysis of a Dykstra-Type Projection Algorithm

Xiaozhou Wang,Ting Kei Pong

DOI: https://doi.org/10.1137/23m1545781

IF: 2.763

2024-02-07

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 563-589, March 2024. Given closed convex sets [math], [math], and some nonzero linear maps [math], [math], of suitable dimensions, the multiset split feasibility problem aims at finding a point in [math] based on computing projections onto [math] and multiplications by [math] and [math]. In this paper, we consider the associated best approximation problem, i.e., the problem of computing projections onto [math]; we refer to this problem as the best approximation problem in multiset split feasibility settings (BA-MSF). We adapt the Dykstra's projection algorithm, which is classical for solving the BA-MSF in the special case when all [math], to solve the general BA-MSF. Our Dykstra-type projection algorithm is derived by applying (proximal) coordinate gradient descent to the Lagrange dual problem, and it only requires computing projections onto [math] and multiplications by [math] and [math] in each iteration. Under a standard relative interior condition and a genericity assumption on the point we need to project, we show that the dual objective satisfies the Kurdyka-Łojasiewicz property with an explicitly computable exponent on a neighborhood of the (typically unbounded) dual solution set when each [math] is [math]-cone reducible for some [math]: this class of sets covers the class of [math]-cone reducible sets, which include all polyhedrons, second-order cone, and the cone of positive semidefinite matrices as special cases. Using this, explicit convergence rate (linear or sublinear) of the sequence generated by the Dykstra-type projection algorithm is derived. Concrete examples are constructed to illustrate the necessity of some of our assumptions.

mathematics, applied
Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Xiangmei Wang,Chong Li,Jinhua Wang,Jen-Chih Yao

DOI: https://doi.org/10.1137/14099961x

IF: 2.763

2015-01-01

SIAM Journal on Optimization

Abstract:We study the convergence issue of the subgradient algorithm for solving the convex feasibility problems in Riemannian manifolds, which was first proposed and analyzed by Bento and Melo [J. Optim. Theory Appl., 152 (2012), pp. 773-785]. The linear convergence property about the subgradient algorithm for solving the convex feasibility problems with the Slater condition in Riemannian manifolds are established, and some step sizes rules are suggested for finite convergence purposes, which are motivated by the work due to De Pierro Iusem [Appl. Math. Optim., 17 (1988), pp. 225-235]. As a by-product, the convergence result of this algorithm is obtained for the convex feasibility problem without the Slater condition assumption. These results extend and/or improve the corresponding known ones in both the Euclidean space and Riemannian manifolds.
Alternating Projection Method for Intersection of Convex Sets, Multi-Agent Consensus Algorithms, and Averaging Inequalities

Proskurnikov, A. V.,Zabarianska, I. S.

DOI: https://doi.org/10.1134/s0965542524700155

2024-06-08

Computational Mathematics and Mathematical Physics

Abstract:The history of the alternating projection method for finding a common point of several convex sets in Euclidean space goes back to the well-known Kaczmarz algorithm for solving systems of linear equations, which was devised in the 1930s and later found wide applications in image processing and computed tomography. An important role in the study of this method was played by I.I. Eremin's, L.M. Bregman's, and B.T. Polyak's works, which appeared nearly simultaneously and contained general results concerning the convergence of alternating projections to a point in the intersection of sets, assuming that this intersection is nonempty. In this paper, we consider a modification of the convex set intersection problem that is related to the theory of multi-agent systems and is called the constrained consensus problem. Each convex set in this problem is associated with a certain agent and, generally speaking, is inaccessible to the other agents. A group of agents is interested in finding a common point of these sets, that is, a point satisfying all the constraints. Distributed analogues of the alternating projection method proposed for solving this problem lead to a rather complicated nonlinear system of equations, the convergence of which is usually proved using special Lyapunov functions. A brief survey of these methods is given, and their relation to the theorem ensuring consensus in a system of averaging inequalities recently proved by the second author is shown (this theorem develops convergence results for the usual method of iterative averaging as applied to the consensus problem).

mathematics, applied,physics, mathematical
PROJECTION ALGORITHMS FOR CONVEX FEASIBILITY PROBLEMS ON HADAMARD MANIFOLDS

Xiangmei Wang,Chong Li,Jen-Chin Yao

2016-01-01

Journal of nonlinear and convex analysis

Abstract:A family of algorithms for solving convex feasibility problems on Hadamard manifolds is studied. Some convergence results and linear convergence results are established, which extend the corresponding ones from Hilbert spaces to Hadamrd manifolds. At last, a numerical example on the Poincare plane is provided to illustrate the theoretical results.
Subgradient Projection Algorithms for Convex Feasibility on Riemannian Manifolds with Lower Bounded Curvatures

X. M. Wang,C. Li,J. C. Yao

DOI: https://doi.org/10.1007/s10957-014-0568-9

2014-01-01

Journal of Optimization Theory and Applications

Abstract:Under the assumption that the sectional curvature of the manifold is bounded from below, we establish convergence result about the cyclic subgradient projection algorithm for convex feasibility problem presented in a paper by Bento and Melo on Riemannian manifolds (J Optim Theory Appl 152, 773–785, 2012). If, additionally, we assume that a Slater type condition is satisfied, then we further show that, without changing the step size, this algorithm terminates in a finite number of iterations. Clearly, our results extend the corresponding ones due to Bento and Melo and, in particular, we solve partially the open problem proposed in the paper by Bento and Melo.
Strong convergence of Bregman projection algorithms for solving split feasibility problems

Liya Liu,Songxiao Li,Bing Tan

DOI: https://doi.org/10.1016/j.cnsns.2024.108374

IF: 4.186

2024-09-30

Communications in Nonlinear Science and Numerical Simulation

Abstract:Bregman distance methods play a key role in solving problems in nonlinear analysis and optimization theory, since the Bregman distance is a useful substitute for the metric. The main purpose of this paper is to investigate two new iterative algorithms based on the Bregman distance and the Bregman projection for solving split feasibility problems in real Hilbert spaces. The algorithms are constructed around these methods: Byrne's CQ method, Polyak's gradient method, Halpern method, and hybrid projection method. The proposed methods involve inertial extrapolation terms and self-adaptive step sizes. We prove that the proposed iterations converge strongly to the Bregman projection of the initial point onto the solution set. Some numerical examples are provided to illustrate the computational effectiveness of our algorithms. The main results extend and improve the recent results related to the split feasibility problem.

mathematics, applied, interdisciplinary applications,mechanics,physics, mathematical, fluids & plasmas
Linear Optimization by Conical Projection

Evgeni Nurminski,Roman Tarasov

2023-09-07

Abstract:This article focuses on numerical efficiency of projection algorithms for solving linear optimization problems. The theoretical foundation for this approach is provided by the basic result that bounded finite dimensional linear optimization problem can be solved by single projection operation on the feasible polyhedron. The further simplification transforms this problem into projection of a special point onto a convex polyhedral cone generated basically by inequalities of the original linear optimization problem.

Optimization and Control
Convergence and Perturbation Resilience of Dynamic String-Averaging Projection Methods

Yair Censor,Alexander J. Zaslavski

DOI: https://doi.org/10.1007/s10589-012-9491-x

2012-06-01

Abstract:We consider the convex feasibility problem (CFP) in Hilbert space and concentrate on the study of string-averaging projection (SAP) methods for the CFP, analyzing their convergence and their perturbation resilience. In the past, SAP methods were formulated with a single predetermined set of strings and a single predetermined set of weights. Here we extend the scope of the family of SAP methods to allow iteration-index-dependent variable strings and weights and term such methods dynamic string-averaging projection (DSAP) methods. The bounded perturbation resilience of DSAP methods is relevant and important for their possible use in the framework of the recently developed superiorization heuristic methodology for constrained minimization problems.

Optimization and Control,Numerical Analysis,Medical Physics
On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Yaohua Hu,Chong Li,Xiaoqi Yang

DOI: https://doi.org/10.1137/140993090

IF: 2.763

2016-01-01

SIAM Journal on Optimization

Abstract:In the present paper, we investigate a linearized proximal algorithm (LPA) for solving a convex composite optimization problem.Each iteration of the LPA is a proximal minimization of the convex composite function with the inner function being linearizedat the current iterate. The LPA has the attractive computational advantage that the solution of each subproblem is a singleton,which avoids the difficulty as in the Gauss--Newton method (GNM) of finding a solution with minimum norm among the set of minimaof its subproblem, while still maintaining the same local convergence rate as that of the GNM. Under the assumptions of localweak sharp minima of order $p$ ($p \in [1,2]$) and a quasi-regularity condition, we establish a local superlinear convergencerate for the LPA. We also propose a globalization strategy for the LPA based on a backtracking line-search and an inexact versionof the LPA. We further apply the LPA to solve a (possibly nonconvex) feasibility problem, as well as a sensor network localizationproblem. Our numerical results illustrate that the LPA meets the demand for an efficient and robust algorithm for the sensor networklocalization problem.
Linear Regularity and Linear Convergence of Projection-Based Methods for Solving Convex Feasibility Problems

Xiaopeng Zhao,Kung Fu Ng,Chong Li,Jen-Chih Yao

DOI: https://doi.org/10.1007/s00245-017-9417-1

2017-01-01

Applied Mathematics & Optimization

Abstract:For a finite/infinite family of closed convex sets with nonempty intersection in Hilbert space, we consider the (bounded) linear regularity property and the linear convergence property of the projection-based methods for solving the convex feasibility problem. Several sufficient conditions are provided to ensure the bounded linear regularity in terms of the interior-point conditions and some finite codimension assumptions. A unified projection method, called Algorithm B-EMOPP, for solving the convex feasibility problem is proposed, and by using the bounded linear regularity, the linear convergence results for this method are established under a new control strategy introduced here.
String-Averaging Projected Subgradient Methods for Constrained Minimization

Y. Censor,A.J. Zaslavski

DOI: https://doi.org/10.48550/arXiv.1308.6378

2013-08-29

Abstract:We consider constrained minimization problems and propose to replace the projection onto the entire feasible region, required in the Projected Subgradient Method (PSM), by projections onto the individual sets whose intersection forms the entire feasible region. Specifically, we propose to perform such projections onto the individual sets in an algorithmic regime of a feasibility-seeking iterative projection method. For this purpose we use the recently developed family of Dynamic String-Averaging Projection (DSAP) methods wherein iteration-index-dependent variable strings and variable weights are permitted. This gives rise to an algorithmic scheme that generalizes, from the algorithmic structural point of view, earlier work of Helou Neto and De Pierro, of Nedić, of Nurminski, and of Ram et al.

Optimization and Control
Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization

Firooz Shahriari-Mehr,Ashkan Panahi

2023-11-08

Abstract:We consider a generic decentralized constrained optimization problem over static, directed communication networks, where each agent has exclusive access to only one convex, differentiable, local objective term and one convex constraint set. For this setup, we propose a novel decentralized algorithm, called DAGP (Double Averaging and Gradient Projection), based on local gradients, projection onto local constraints, and local averaging. We achieve global optimality through a novel distributed tracking technique we call distributed null projection. Further, we show that DAGP can be used to solve unconstrained problems with non-differentiable objective terms with a problem reduction scheme. Assuming only smoothness of the objective terms, we study the convergence of DAGP and establish sub-linear rates of convergence in terms of feasibility, consensus, and optimality, with no extra assumption (e.g. strong convexity). For the analysis, we forego the difficulties of selecting Lyapunov functions by proposing a new methodology of convergence analysis in optimization problems, which we refer to as aggregate lower-bounding. To demonstrate the generality of this method, we also provide an alternative convergence proof for the standard gradient descent algorithm with smooth functions. Finally, we present numerical results demonstrating the effectiveness of our proposed method in both constrained and unconstrained problems. In particular, we propose a distributed scheme by DAGP for the optimal transport problem with superior performance and speed.

Optimization and Control,Distributed, Parallel, and Cluster Computing,Multiagent Systems
Random Multi-Constraint Projection: Stochastic Gradient Methods for Convex Optimization with Many Constraints

Mengdi Wang,Yichen Chen,Jialin Liu,Yuantao Gu

DOI: https://doi.org/10.48550/arxiv.1511.03760

2015-01-01

Abstract:Consider convex optimization problems subject to a large number of constraints. We focus on stochastic problems in which the objective takes the form of expected values and the feasible set is the intersection of a large number of convex sets. We propose a class of algorithms that perform both stochastic gradient descent and random feasibility updates simultaneously. At every iteration, the algorithms sample a number of projection points onto a randomly selected small subsets of all constraints. Three feasibility update schemes are considered: averaging over random projected points, projecting onto the most distant sample, projecting onto a special polyhedral set constructed based on sample points. We prove the almost sure convergence of these algorithms, and analyze the iterates' feasibility error and optimality error, respectively. We provide new convergence rate benchmarks for stochastic first-order optimization with many constraints. The rate analysis and numerical experiments reveal that the algorithm using the polyhedral-set projection scheme is the most efficient one within known algorithms.
O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Lijun Zhang,Tianbao Yang,Rong Jin,Xiaofei He

DOI: https://doi.org/10.48550/arxiv.1304.0740

2013-01-01

Abstract:Traditional algorithms for stochastic optimization require projecting the solution at each iteration into a given domain to ensure its feasibility. When facing complex domains, such as positive semi-definite cones, the projection operation can be expensive, leading to a high computational cost per iteration. In this paper, we present a novel algorithm that aims to reduce the number of projections for stochastic optimization. The proposed algorithm combines the strength of several recent developments in stochastic optimization, including mini-batch, extra-gradient, and epoch gradient descent, in order to effectively explore the smoothness and strong convexity. We show, both in expectation and with a high probability, that when the objective function is both smooth and strongly convex, the proposed algorithm achieves the optimal O(1/T) rate of convergence with only O(log T) projections. Our empirical study verifies the theoretical result.
Averaging on the Bures-Wasserstein manifold: dimension-free convergence of gradient descent

Jason M. Altschuler,Sinho Chewi,Patrik Gerber,Austin J. Stromme

2023-10-31

Abstract:We study first-order optimization algorithms for computing the barycenter of Gaussian distributions with respect to the optimal transport metric. Although the objective is geodesically non-convex, Riemannian GD empirically converges rapidly, in fact faster than off-the-shelf methods such as Euclidean GD and SDP solvers. This stands in stark contrast to the best-known theoretical results for Riemannian GD, which depend exponentially on the dimension. In this work, we prove new geodesic convexity results which provide stronger control of the iterates, yielding a dimension-free convergence rate. Our techniques also enable the analysis of two related notions of averaging, the entropically-regularized barycenter and the geometric median, providing the first convergence guarantees for Riemannian GD for these problems.

Optimization and Control,Machine Learning
How averaged is the projection?

Shuang Song

2023-12-24

Abstract:Projection operators are important in Analysis, Optimization and Algorithm. It is well known that these operators are firmly nonexpansive. In this paper, we provide an exact result that sharpens this well-known result. We develop the theory of averaged operators and provide a lower bound. We give a result on the avergedness of operator compositions. We also provide some nonlinear examples to illustrate our results.

Functional Analysis,Numerical Analysis,Optimization and Control
Algorithms and Convergence Results of Projection Methods for Inconsistent Feasibility Problems: A Review

Yair Censor,Maroun Zaknoon

DOI: https://doi.org/10.48550/arXiv.1802.07529

2018-04-27

Abstract:The convex feasibility problem (CFP) is to find a feasible point in the intersection of finitely many convex and closed sets. If the intersection is empty then the CFP is inconsistent and a feasible point does not exist. However, algorithmic research of inconsistent CFPs exists and is mainly focused on two directions. One is oriented toward defining other solution concepts that will apply, such as proximity function minimization wherein a proximity function measures in some way the total violation of all constraints. The second direction investigates the behavior of algorithms that are designed to solve a consistent CFP when applied to inconsistent problems. This direction is fueled by situations wherein one lacks a priori information about the consistency or inconsistency of the CFP or does not wish to invest computational resources to get hold of such knowledge prior to running his algorithm. In this paper we bring under one roof and telegraphically review some recent works on inconsistent CFPs.

Optimization and Control
Angle criteria for uniform convergence of averaged projections and cyclic or random products of projections

Izhar Oppenheim

DOI: https://doi.org/10.1007/s11856-017-1618-4

2017-11-24

Israel Journal of Mathematics

Abstract:We apply a new notion of angle between projections to deduce criteria for uniform convergence results of the alternating projections method under several different settings: averaged projections, cyclic products, quasiperiodic products and random products.

mathematics

Convergence of weighted averages of relaxed projections

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

Convergence Rate Analysis of a Dykstra-Type Projection Algorithm

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

Alternating Projection Method for Intersection of Convex Sets, Multi-Agent Consensus Algorithms, and Averaging Inequalities

PROJECTION ALGORITHMS FOR CONVEX FEASIBILITY PROBLEMS ON HADAMARD MANIFOLDS

Subgradient Projection Algorithms for Convex Feasibility on Riemannian Manifolds with Lower Bounded Curvatures

Strong convergence of Bregman projection algorithms for solving split feasibility problems

Linear Optimization by Conical Projection

Convergence and Perturbation Resilience of Dynamic String-Averaging Projection Methods

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Linear Regularity and Linear Convergence of Projection-Based Methods for Solving Convex Feasibility Problems

String-Averaging Projected Subgradient Methods for Constrained Minimization

Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization

Random Multi-Constraint Projection: Stochastic Gradient Methods for Convex Optimization with Many Constraints

O(log T) Projections for Stochastic Optimization of Smooth and Strongly Convex Functions

Averaging on the Bures-Wasserstein manifold: dimension-free convergence of gradient descent

How averaged is the projection?

Algorithms and Convergence Results of Projection Methods for Inconsistent Feasibility Problems: A Review

Angle criteria for uniform convergence of averaged projections and cyclic or random products of projections