Variable metric algorithms driven by averaged operators

Lilian E. Glaudin

DOI: https://doi.org/10.48550/arXiv.1807.04027

2018-07-11

Abstract:The convergence of a new general variable metric algorithm based on compositions of averaged operators is established. Applications to monotone operator splitting are presented.

Optimization and Control

What problem does this paper attempt to address?

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Jinming Xu,Ying Sun,Ye Tian,Gesualdo Scutari

DOI: https://doi.org/10.1109/camsap45676.2019.9022451

2019-01-01

Abstract:We study distributed composite optimization over networks: agents minimize the sum of a smooth (strongly) convex function, the agents' sum-utility, plus a non-smooth (extended-valued) convex one. We propose a general algorithmic framework for such a class of problems and provide a unified convergence analysis leveraging the theory of operator splitting. Our results unify several approaches proposed in the literature of distributed optimization for special instances of our formulation. Distinguishing features of our scheme are: (i) when the agents' functions are strongly convex, the algorithm converges at a linear rate, whose dependencies on the agents' functions and the network topology are decoupled, matching the typical rates of centralized optimization; (ii) the step-size does not depend on the network parameters but only on the optimization ones; and (iii) the algorithm can adjust the ratio between the number of communications and computations to achieve the same rate of the centralized proximal gradient scheme (in terms of computations). This is the first time that a distributed algorithm applicable to composite optimization enjoys such properties.
Almost-Surely Convergent Randomly Activated Monotone Operator Splitting Methods

Patrick L. Combettes,Javier I. Madariaga

2024-11-14

Abstract:We propose stochastic splitting algorithms for solving large-scale composite inclusion problems involving monotone and linear operators. They activate at each iteration blocks of randomly selected resolvents of monotone operators and, unlike existing methods, achieve almost sure convergence of the iterates to a solution without any regularity assumptions or knowledge of the norms of the linear operators. Applications to image recovery and machine learning are provided.

Optimization and Control
Convergence of Splitting Algorithms for the Sum of Two Accretive Operators with Applications

Xiaolong Qin,Sun Young Cho,Lin Wang

DOI: https://doi.org/10.1186/1687-1812-2014-166

2014-01-01

Fixed Point Theory and Applications

Abstract:We study a splitting algorithm for problems involving the sum of two accretive operators. We prove the strong convergence of the algorithm. Applications to variational inequality, fixed point, equilibrium, and minimization problems are provided.
Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Jinming Xu,Ye Tian,Ying Sun,Gesualdo Scutari

DOI: https://doi.org/10.1109/tsp.2021.3086579

IF: 4.875

2021-01-01

IEEE Transactions on Signal Processing

Abstract:We study distributed composite optimization over networks: agents minimize a sum of smooth (strongly) convex functions, the agents' sum-utility, plus a nonsmooth (extended-valued) convex one. We propose a general unified algorithmic framework for such a class of problems and provide a unified convergence analysis leveraging the theory of operator splitting. Distinguishing features of our scheme are: (i) When the agents' functions are strongly convex, the algorithm converges at a linear rate, whose dependence on the agents' functions and network topology is decoupled, matching the typical rates of centralized optimization; the rate expression improves on existing results; (ii) When the objective function is convex (but not strongly convex), similar separation as in (i) is established for the coefficient of the proved sublinear rate; (iii) The algorithm can adjust the ratio between the number of communications and computations to achieve a rate (in terms of computations) independent on the network connectivity; and (iv) A by-product of our analysis is a tuning recommendation for several existing (non accelerated) distributed algorithms yielding the fastest provably (worst-case) convergence rate. This is the first time that a general distributed algorithmic framework applicable to composite optimization enjoys all such properties.
New averaged type algorithms for solving split common fixed-point problem for demicontractive mappings

Vasile Berinde,Khairul Saleh

2024-04-05

Abstract:In this paper we propose new averaged iterative algorithms designed for solving a split common fixed-point problem in the class of demicontractive mappings. The algorithms are obtained by inserting an averaged term into the algorithms used in [Li, R. and He, Z., A new iterative algorithm for split solution problems of quasi-nonexpansive mappings {\it J. Inequal. Appl.} {\bf 131} (2015), 1--12.] for solving the same problem but in the class of quasi-nonexpansive mappings, which is a subclass of demicontractive mappings. Basically, our investigation is based on the embedding of demicontractive operators in the class of quasi-nonexpansive operators by means of averaged mappings. For the considered algorithms we prove weak and strong convergence theorems in the setting of a real Hilbert space and also provide examples to show that our results are effective generalizations of existing results in literature.

General Mathematics
A Unified Algorithmic Framework for Distributed Composite Optimization.

Jinming Xu,Ye Tian,Ying Sun,Gesualdo Scutari

DOI: https://doi.org/10.1109/CDC42340.2020.9303791

2020-01-01

Abstract:We study distributed composite optimization over networks: agents minimize a sum of smooth (strongly) convex functions-the agents' sum-utility-plus a nonsmooth (extended-valued) convex one. We propose a general unified algorithmic framework for such a class of problems, whose unified convergence analysis leverages the theory of operator splitting. Distinguishing features of our scheme are: (i) When the agents' functions are strongly convex, the algorithm converges at a linear rate, whose dependence on the agents' functions and network topology is decoupled, matching the typical rates of centralized optimization; the rate expression improves on existing results; (ii) When the objective function is convex (but not strongly convex), similar separation as in (i) is established for the coefficient of the proved sublinear rate; and (iii) A byproduct of our analysis is a tuning recommendation for several existing (non-accelerated) distributed algorithms yielding the fastest provable (worst-case) convergence rate. This is the first time that a general distributed algorithmic framework applicable to composite optimization enjoys all such properties.
A strongly convergent algorithm for solving split equality problems beyond monotonicity

Oluwatosin Temitope Mewomo,Victor Amarachi Uzor,Aviv Gibali

DOI: https://doi.org/10.1007/s40314-024-02829-w

2024-07-14

Computational and Applied Mathematics

Abstract:In this paper, we focus on some split inverse problems, namely the split equality variational inequalities and common fixed point problems, and combine various operator theory techniques to establish minimum-norm strong convergence for our proposed method. We present two strong convergent results with (and without) reference to the monotonicity property of the cost operators. Our convergence analyses assume very mild conditions and thus generalize and extend recent related results in the literature. Furthermore, several numerical examples illustrate the practical potentials and advantages of our proposed algorithm.

mathematics, applied
Strong Convergence of A Splitting Algorithm for Treating Monotone Operators

Sun Young Cho,Xiaolong Qin,Lin Wang

DOI: https://doi.org/10.1186/1687-1812-2014-94

2014-01-01

Fixed Point Theory and Applications

Abstract:In this paper, we investigate a splitting algorithm for treating monotone operators. Strong convergence theorems are established in the framework of Hilbert spaces.
Frugal Splitting Operators: Representation, Minimal Lifting, and Convergence

Martin Morin,Sebastian Banert,Pontus Giselsson

DOI: https://doi.org/10.1137/22m1531105

IF: 2.763

2024-05-01

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1595-1621, June 2024. We investigate frugal splitting operators for finite sum monotone inclusion problems. These operators utilize exactly one direct or resolvent evaluation of each operator of the sum, and the splitting operator's output is dictated by linear combinations of these evaluations' inputs and outputs. To facilitate analysis, we introduce a novel representation of frugal splitting operators via a generalized primal-dual resolvent. The representation is characterized by an index and four matrices, and we provide conditions on these that ensure equivalence between the classes of frugal splitting operators and generalized primal-dual resolvents. Our representation paves the way for new results regarding lifting numbers and the development of a unified convergence analysis for frugal splitting operator methods, contingent on the directly evaluated operators being cocoercive. The minimal lifting number is [math] where [math] is the number of monotone operators and [math] is the number of direct evaluations in the splitting. Notably, this lifting number is achievable only if the first and last operator evaluations are resolvent evaluations. These results generalize the minimal lifting results by Ryu and by Malitsky and Tam that consider frugal resolvent splittings. Building on our representation, we delineate a constructive method to design frugal splitting operators, exemplified in the design of a novel, convergent, and parallelizable frugal splitting operator with minimal lifting.

mathematics, applied
The Geometry of Monotone Operator Splitting Methods

Patrick L. Combettes

2024-01-30

Abstract:We propose a geometric framework to describe and analyze a wide array of operator splitting methods for solving monotone inclusion problems. The initial inclusion problem, which typically involves several operators combined through monotonicity-preserving operations, is seldom solvable in its original form. We embed it in an auxiliary space, where it is associated with a surrogate monotone inclusion problem with a more tractable structure and which allows for easy recovery of solutions to the initial problem. The surrogate problem is solved by successive projections onto half-spaces containing its solution set. The outer approximation half-spaces are constructed by using the individual operators present in the model separately. This geometric framework is shown to encompass traditional methods as well as state-of-the-art asynchronous block-iterative algorithms, and its flexible structure provides a pattern to design new ones.

Optimization and Control
The splitting algorithms by Ryu, by Malitsky-Tam, and by Campoy applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

Heinz H. Bauschke,Shambhavi Singh,Xianfu Wang

DOI: https://doi.org/10.48550/arXiv.2203.03832

2022-12-20

Abstract:Finding a zero of a sum of maximally monotone operators is a fundamental problem in modern optimization and nonsmooth analysis. Assuming that the resolvents of the operators are available, this problem can be tackled with the Douglas-Rachford algorithm. However, when dealing with three or more operators, one must work in a product space with as many factors as there are operators. In groundbreaking recent work by Ryu and by Malitsky and Tam, it was shown that the number of factors can be reduced by one. A similar reduction was achieved recently by Campoy through a clever reformulation originally proposed by Kruger. All three splitting methods guarantee weak convergence to some solution of the underlying sum problem; strong convergence holds in the presence of uniform monotonicity. In this paper, we provide a case study when the operators involved are normal cone operators of subspaces and the solution set is thus the intersection of the subspaces. Even though these operators lack strict convexity, we show that striking conclusions are available in this case: strong (instead of weak) convergence and the solution obtained is (not arbitrary but) the projection onto the intersection. Numerical experiments to illustrate our results are also provided.

Optimization and Control
Global Convergence of an Inexact Operator Splitting Method for Monotone Variational Inequalities

Zhili Ge,Gang Qian,Deren Han

DOI: https://doi.org/10.3934/jimo.2011.7.1013

2011-01-01

Journal of Industrial and Management Optimization

Abstract:Recently, Han (Han D, Inexact operator splitting methods with self-adaptive strategy for variational inequality problems, Journal of Optimization Theory and Applications 132, 227-243 (2007)) proposed an inexact operator splitting method for solving variational inequality problems. It has advantage over the classical operator splitting method of Douglas-Peaceman-Rachford-Varga operator splitting methods (DPRV methods) and some of their variants, since it adopts a very flexible approximate rule in solving the subproblem in each iteration. However, its convergence is established under some what stringent condition that the underlying mapping F is strongly monotone. In this paper, we mainly establish the global convergence of the method under weaker condition that the underlying mapping F is monotone, which extends the fields of applications of the method relatively. Some numerical results are also presented to illustrate the method.
Convergence Analysis of an Iterative Algorithm for Monotone Operators

Sun Young Cho,Wenling Li,Shin Min Kang

DOI: https://doi.org/10.1186/1029-242x-2013-199

2013-01-01

Journal of Inequalities and Applications

Abstract:In this paper, an iterative algorithm is proposed to study some nonlinear operators which are inverse-strongly monotone, maximal monotone, and strictly pseudocontractive. Strong convergence of the proposed iterative algorithm is obtained in the framework of Hilbert spaces.
Accelerated Gradient and Skew-Symmetric Splitting Methods for a Class of Monotone Operator Equations

Long Chen,Jingrong Wei

2023-03-16

Abstract:A class of monotone operator equations, which can be decomposed into sum of a gradient of a strongly convex function and a linear and skew-symmetric operator, is considered in this work. Based on discretization of the generalized gradient flow, gradient and skew-symmetric splitting (GSS) methods are proposed and proved to convergent in linear rate. To further accelerate the convergence, an accelerated gradient flow is proposed and accelerated gradient and skew-symmetric splitting (AGSS) methods are developed, which extends the acceleration among the existing works on the convex minimization to a more general class of monotone operator equations. In particular, when applied to smooth saddle point systems with bilinear coupling, an accelerated transformed primal-dual (ATPD) method is proposed and shown to achieve linear rates with optimal lower iteration complexity.

Optimization and Control
The splitting algorithms by Ryu and by Malitsky-Tam applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

Heinz H. Bauschke,Shambhavi Singh,Xianfu Wang

DOI: https://doi.org/10.48550/arXiv.2109.11072

2021-09-23

Abstract:Finding a zero of a sum of maximally monotone operators is a fundamental problem in modern optimization and nonsmooth analysis. Assuming that resolvents of the operators are available, this problem can be tackled with the Douglas-Rachford algorithm. However, when dealing with three or more operators, one must work in a product space with as many factors as there are operators. In groundbreaking recent work by Ryu and by Malitsky and Tam, it was shown that the number of factors can be reduced by one. These splitting methods guarantee weak convergence to some solution of the underlying sum problem; strong convergence holds in the presence of uniform monotonicity. In this paper, we provide a case study when the operators involved are normal cone operators of subspaces and the solution set is thus the intersection of the subspaces. Even though these operators lack strict convexity, we show that striking conclusions are available in this case: strong (instead of weak) convergence and the solution obtained is (not arbitrary but) the projection onto the intersection. Numerical experiments to illustrate our results are also provided

Optimization and Control
On compositions of special cases of Lipschitz continuous operators

Pontus Giselsson,Walaa M. Moursi

DOI: https://doi.org/10.48550/arXiv.1912.13165

2019-12-31

Abstract:Many iterative optimization algorithms involve compositions of special cases of Lipschitz continuous operators, namely firmly nonexpansive, averaged and nonexpansive operators. The structure and properties of the compositions are of particular importance in the proofs of convergence of such algorithms. In this paper, we systematically study the compositions of further special cases of Lipschitz continuous operators. Applications of our results include compositions of scaled conically nonexpansive mappings, as well as the Douglas--Rachford and forward-backward operators, when applied to solve certain structured monotone inclusion and optimization problems. Several examples illustrate and tighten our conclusions.

Optimization and Control
An Alternating Procedure for Operators on Uniformly Convex and Uniformly Smooth Banach-Spaces

ZB XU,GF ROACH

DOI: https://doi.org/10.2307/2048574

IF: 0.9709

1991-01-01

Proceedings of the American Mathematical Society

Abstract:Let $X$ be a real uniformly convex and uniformly smooth Banach space. For any $1 < p < \infty ,{J_p},J_p^ *$ respectively denote the duality mapping with gauge function $\varphi (t) = {t^{p - 1}}$ from $X$ onto ${X^ * }$ and ${X^*}$ onto $X$. If $T:X \to X$ is a bounded linear operator, then $M(T):X \to X$ is the mapping defined by $M(T) = J_q^ * {T^ * }{J_p}T$, where ${T^ * }:{X^ * } \to {X^ * }$ is the adjoint of $T$ and $q = {(p - 1)^{ - 1}}p$. It is proved that if ${T_n}$ is a sequence of operators on $X$ such that $\left \| {{T_n}} \right \| \leq 1$ for all $n$, then $M({T_n}, \ldots ,{T_1})x$ strongly converges in $X$ for any $x \in X$, with an estimate of the rate of convergence: \[ ||M({T_n} \cdots {T_1})x - M(x)|| \leq \sigma (x)||x||\psi (1 - {(m(x)/||{T_n} \cdots {T_1}x||)^p}),\], where $M(x) = {\lim _{n \to \infty }}M({T_n} \cdots {T_1})x,m(x) = {\lim _{n \to \infty }}\left \| {{T_n} \cdots {T_1}x} \right \|$, and $\sigma :X \to {R^ + },\psi :{R^ + } \to {R^ + }$ are definite, strictly increasing positive functions. The result obtained generalizes and improves on the theorem offered recently by Akcoglu and Sucheston [1].
On the convergence properties of non-Euclidean extragradient methods for variational inequalities with generalized monotone operators

Cong D. Dang,Guanghui Lan

DOI: https://doi.org/10.1007/s10589-014-9673-9

2014-06-24

Computational Optimization and Applications

Abstract:In this paper, we study a class of generalized monotone variational inequality (GMVI) problems whose operators are not necessarily monotone (e.g., pseudo-monotone). We present non-Euclidean extragradient (N-EG) methods for computing approximate strong solutions of these problems, and demonstrate how their iteration complexities depend on the global Lipschitz or Hölder continuity properties for their operators and the smoothness properties for the distance generating function used in the N-EG algorithms. We also introduce a variant of this algorithm by incorporating a simple line-search procedure to deal with problems with more general continuous operators. Numerical studies are conducted to illustrate the significant advantages of the developed algorithms over the existing ones for solving large-scale GMVI problems.

mathematics, applied,operations research & management science
How averaged is the projection?

Shuang Song

2023-12-24

Abstract:Projection operators are important in Analysis, Optimization and Algorithm. It is well known that these operators are firmly nonexpansive. In this paper, we provide an exact result that sharpens this well-known result. We develop the theory of averaged operators and provide a lower bound. We give a result on the avergedness of operator compositions. We also provide some nonlinear examples to illustrate our results.

Functional Analysis,Numerical Analysis,Optimization and Control
Forward Primal-Dual Half-Forward Algorithm for Splitting Four Operators

Fernando Roldán

2024-09-26

Abstract:In this article, we propose a splitting algorithm to find zeros of the sum of four maximally monotone operators in real Hilbert spaces. In particular, we consider a Lipschitzian operator, a cocoercive operator, and a linear composite term. In the case when the Lipschitzian operator is absent, our method reduces to the Condat-Vũ algorithm. On the other hand, when the linear composite term is absent, the algorithm reduces to the Forward-Backward-Half-Forward algorithm (FBHF). Additionally, in each case, the set of step-sizes that guarantee the weak convergence of those methods are recovered. Therefore, our algorithm can be seen as a generalization of Condat-Vũ and FBHF. Moreover, we propose extensions and applications of our method in multivariate monotone inclusions and saddle point problems. Finally, we present a numerical experiment in image deblurring problems.

Optimization and Control

Variable metric algorithms driven by averaged operators

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Almost-Surely Convergent Randomly Activated Monotone Operator Splitting Methods

Convergence of Splitting Algorithms for the Sum of Two Accretive Operators with Applications

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

New averaged type algorithms for solving split common fixed-point problem for demicontractive mappings

A Unified Algorithmic Framework for Distributed Composite Optimization.

A strongly convergent algorithm for solving split equality problems beyond monotonicity

Strong Convergence of A Splitting Algorithm for Treating Monotone Operators

Frugal Splitting Operators: Representation, Minimal Lifting, and Convergence

The Geometry of Monotone Operator Splitting Methods

The splitting algorithms by Ryu, by Malitsky-Tam, and by Campoy applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

Global Convergence of an Inexact Operator Splitting Method for Monotone Variational Inequalities

Convergence Analysis of an Iterative Algorithm for Monotone Operators

Accelerated Gradient and Skew-Symmetric Splitting Methods for a Class of Monotone Operator Equations

The splitting algorithms by Ryu and by Malitsky-Tam applied to normal cones of linear subspaces converge strongly to the projection onto the intersection

On compositions of special cases of Lipschitz continuous operators

An Alternating Procedure for Operators on Uniformly Convex and Uniformly Smooth Banach-Spaces

On the convergence properties of non-Euclidean extragradient methods for variational inequalities with generalized monotone operators

How averaged is the projection?

Forward Primal-Dual Half-Forward Algorithm for Splitting Four Operators