Comparison of Purely Greedy and Orthogonal Greedy Algorithm

Vishnevetskiy, K. S.

DOI: https://doi.org/10.1134/s0001434624010048

2024-04-23

Mathematical Notes

Abstract:Conditions for a dictionary in a Hilbert space are obtained that are necessary or sufficient for the coincidence of purely greedy and orthogonal greedy algorithms with respect to this dictionary.

mathematics

What problem does this paper attempt to address?

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Jonathan W. Siegel,Jinchao Xu

DOI: https://doi.org/10.1109/tit.2022.3147984

IF: 2.5

2022-05-01

IEEE Transactions on Information Theory

Abstract:We analyze the orthogonal greedy algorithm when applied to dictionaries whose convex hull has small entropy. We show that if the metric entropy of the convex hull of decays at a rate of for , then the orthogonal greedy algorithm converges at the same rate on the variation space of . This improves upon the well-known convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of .

computer science, information systems,engineering, electrical & electronic
Projection Greedy Algorithm

P. A. Borodin,S. V. Konyagin

DOI: https://doi.org/10.1134/S0001434621070026

2021-08-24

Mathematical Notes

Abstract:We introduce and study a new type of greedy algorithm, namely, projection greedy algorithms with respect to a given dictionary in a Hilbert space. We prove that these algorithms converge and estimate the rate of convergence for initial elements from the convex hull of the dictionary. Several specific examples of dictionaries are used to compare the introduced algorithms with orthogonal greedy algorithms.

mathematics
On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.1003.5349

2010-03-28

Numerical Analysis

Abstract:We show that Orthogonal Greedy Algorithms (Orthogonal Matching Pursuit) provides almost optimal approximation on the first [1/(20M)] steps for M-coherent dictionaries
On Greedy Algorithms with bounded cumulative coherence

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.0911.1500

2009-11-08

Abstract:We discuss the upper and lower estimates for the rate of convergence of Pure and Orthogonal Greedy Algorithms for dictionary with bounded cumulative coherence.

Numerical Analysis
Biorthogonal Greedy Algorithms in convex optimization

A.V. Dereventsov,V.N. Temlyakov

DOI: https://doi.org/10.1016/j.acha.2022.05.001

IF: 2.974

2022-09-01

Applied and Computational Harmonic Analysis

Abstract:The study of greedy approximation in the context of convex optimization is becoming a promising research direction as greedy algorithms are actively being employed to construct sparse minimizers for convex functions with respect to given sets of elements. In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms for the minimization of convex functions on Banach spaces. Specifically, we define the class of Weak Biorthogonal Greedy Algorithms for convex optimization that contains a wide range of greedy algorithms. We analyze the introduced class of algorithms and establish the properties of convergence, rate of convergence, and numerical stability, which is understood in the sense that the steps of the algorithm are allowed to be performed not precisely but with controlled computational inaccuracies. We show that the following well-known algorithms for convex optimization — the Weak Chebyshev Greedy Algorithm (co) and the Weak Greedy Algorithm with Free Relaxation (co) — belong to this class, and introduce a new algorithm — the Rescaled Weak Relaxed Greedy Algorithm (co). Presented numerical experiments demonstrate the practical performance of the aforementioned greedy algorithms in the setting of convex minimization as compared to optimization with regularization, which is the conventional approach of constructing sparse minimizers.

mathematics, applied
Super greedy type algorithms

Entao Liu,Vladimir N. Temlyakov

DOI: https://doi.org/10.1007/s10444-011-9220-5

2011-09-30

Advances in Computational Mathematics

Abstract:We study greedy-type algorithms such that at a greedy step we pick several dictionary elements contrary to a single dictionary element in standard greedy algorithms. We call such greedy algorithms super greedy type algorithms. The super greedy type algorithms are computationally simpler than their analogues from the standard greedy algorithms. In this article, we propose the Weak Super Greedy Algorithm (WSGA) and the Weak Orthogonal Super Greedy Algorithm with Thresholding (WOSGAT). Their performance (rate of convergence) are studied as well under M-coherent dictionaries.

mathematics, applied
Greedy algorithms: a review and open problems

Andrea García

2024-08-16

Abstract:Greedy algorithms are a fundamental category of algorithms in mathematics and computer science, characterized by their iterative, locally optimal decision-making approach, which aims to find global optima. In this review, we will discuss two greedy algorithms. First, we will talk about the so-called Relaxed Greedy Algorithm in the context of dictionaries in Hilbert spaces analyzing the optimality of definition of this algorithm and, next, we give a general overview of the Thresholding Greedy Algorithm and the Chebyshev Thresholding Greedy Algorithm with regard to bases in p-Banach spaces with $0 < p \leq 1$. In both cases, we pose some questions for future research.

Functional Analysis
Greedy expansions with prescribed coefficients in Hilbert spaces for special classes of dictionaries

Alessandro Oliaro,Luca Tomatis,Albert R. Valiullin,Artur R. Valiullin

2023-07-05

Abstract:Greedy expansions with prescribed coefficients have been introduced by V. N. Temlyakov in the frame of Banach spaces. The idea is to choose a sequence of fixed (real) coefficients $\{c_n\}_{n=1}^\infty$ and a fixed set of elements (dictionary) of the Banach space; then, under suitable conditions on the coefficients and the dictionary, it is possible to expand all the elements of the Banach space in series that contain only the fixed coefficients and the elements of the dictionary. In Hilbert spaces the convergence of greedy algorithm with prescribed coefficients is characterized, in the sense that there are necessary and sufficient conditions on the coefficients in order that the algorithm is convergent for all the dictionaries. This paper is concerned with the question if such conditions can be weakened for particular classes of spaces or dictionaries; we prove that this is the case for finite dimensional spaces, and for some classes of dictionaries related to orthonormal sequences in infinite dimensional spaces.

Functional Analysis
A unified way of analyzing some greedy algorithms

Anton Dereventsov,Vladimir Temlyakov

DOI: https://doi.org/10.1016/j.jfa.2019.108286

2019-06-05

Abstract:In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms in Banach spaces. We define a class of the Weak Biorthogonal Greedy Algorithms that contains a wide range of greedy algorithms. In particular, we show that the following well-known algorithms --- the Weak Chebyshev Greedy Algorithm and the Weak Greedy Algorithm with Free Relaxation --- belong to this class. We investigate the properties of convergence, rate of convergence, and numerical stability of the Weak Biorthogonal Greedy Algorithms. Numerical stability is understood in the sense that the steps of the algorithm are allowed to be performed with controlled computational inaccuracies. We carry out a thorough analysis of the connection between the magnitude of those inaccuracies and the convergence properties of the algorithm. To emphasize the advantage of the proposed approach, we introduce here a new greedy algorithm --- the Rescaled Weak Relaxed Greedy Algorithm --- from the above class, and derive the convergence results without analyzing the algorithm explicitly. Additionally, we explain how the proposed approach can be extended to some other types of greedy algorithms.

Numerical Analysis
Entropy-based convergence rates of greedy algorithms

Yuwen Li,Jonathan Siegel

DOI: https://doi.org/10.1142/S0218202524500143

2024-10-27

Abstract:We present convergence estimates of two types of greedy algorithms in terms of the metric entropy of underlying compact sets. In the first part, we measure the error of a standard greedy reduced basis method for parametric PDEs by the metric entropy of the solution manifold in Banach spaces. This contrasts with the classical analysis based on the Kolmogorov n-widths and enables us to obtain direct comparisons between the greedy algorithm error and the entropy numbers, where the multiplicative constants are explicit and simple. The entropy-based convergence estimate is sharp and improves upon the classical width-based analysis of reduced basis methods for elliptic model problems. In the second part, we derive a novel and simple convergence analysis of the classical orthogonal greedy algorithm for nonlinear dictionary approximation using the metric entropy of the symmetric convex hull of the dictionary. This also improves upon existing results by giving a direct comparison between the algorithm error and the metric entropy.

Numerical Analysis
On the rate of convergence of greedy algorithms

V.N. Temlyakov

2023-04-13

Abstract:We prove some results on the rate of convergence of greedy algorithms, which provide expansions. We consider both the case of Hilbert spaces and the more general case of Banach spaces. The new ingredient of the paper is that we bound the error of approximation by the product of both norms -- the norm of $f$ and the $A_1$-norm of $f$. Typically, only the $A_1$-norm of $f$ is used. In particular, we establish that some greedy algorithms (Pure Greedy Algorithm (PGA) and its generalizations) are as good as the Orthogonal Greedy Algorithm (OGA) in this new sense of the rate of convergence, while it is known that the PGA is much worth than the OGA in the standard sense.

Numerical Analysis,Functional Analysis
Greed is Good: Algorithmic Results for Sparse Approximation

J.A. Tropp

DOI: https://doi.org/10.1109/tit.2004.834793

IF: 2.5

2004-10-01

IEEE Transactions on Information Theory

Abstract:This article presents new results on using a greedy algorithm, orthogonal matching pursuit (OMP), to solve the sparse approximation problem over redundant dictionaries. It provides a sufficient condition under which both OMP and Donoho's basis pursuit (BP) paradigm can recover the optimal representation of an exactly sparse signal. It leverages this theory to show that both OMP and BP succeed for every sparse input signal from a wide class of dictionaries. These quasi-incoherent dictionaries offer a natural generalization of incoherent dictionaries, and the cumulative coherence function is introduced to quantify the level of incoherence. This analysis unifies all the recent results on BP and extends them to OMP. Furthermore, the paper develops a sufficient condition under which OMP can identify atoms from an optimal approximation of a nonsparse signal. From there, it argues that OMP is an approximation algorithm for the sparse problem over a quasi-incoherent dictionary. That is, for every input signal, OMP calculates a sparse approximant whose error is only a small factor worse than the minimal error that can be attained with the same number of terms.

computer science, information systems,engineering, electrical & electronic
Greedy Approximation in Convex Optimization

V. N. Temlyakov

DOI: https://doi.org/10.1007/s00365-014-9272-0

2015-01-21

Constructive Approximation

Abstract:We study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of a few elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. The problem of approximation of a given element of a Banach space by linear combinations of elements from a given system (dictionary) is well studied in nonlinear approximation theory. At first glance, the settings of approximation and optimization problems are very different. In the approximation problem, an element is given and our task is to find a sparse approximation of it. In optimization theory, an energy function is given and we should find an approximate sparse solution to the minimization problem. It turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular the greedy approximation technique, can be adjusted for finding a sparse solution of an optimization problem.

mathematics
Efficient and provably convergent randomized greedy algorithms for neural network optimization

Jinchao Xu,Xiaofeng Xu

2024-09-15

Abstract:Greedy algorithms have been successfully analyzed and applied in training neural networks for solving variational problems, ensuring guaranteed convergence orders. However, their practical applicability is limited due to the subproblems, which often require an exhaustive search over a discrete dictionary and incur significant computational costs. This limitation becomes critical, especially in high-dimensional problems. In this paper, we propose a more practical approach of randomly discretizing the dictionary at each iteration of the greedy algorithm. We quantify the required size of the randomized discrete dictionary and prove that, with high probability, the proposed algorithm realizes a weak greedy algorithm, achieving optimal convergence orders. Through numerous numerical experiments, we demonstrate the advantage of using randomized discrete dictionaries over a deterministic one by showing orders of magnitude reductions in the size of the discrete dictionary, particularly in higher dimensions.

Numerical Analysis
Greedy Algorithms for Reduced Bases in Banach Spaces

Ronald DeVore,Guergana Petrova,Przemyslaw Wojtaszczyk

DOI: https://doi.org/10.48550/arXiv.1204.2290

2012-04-11

Abstract:Given a Banach space X and one of its compact sets F, we consider the problem of finding a good n dimensional space X_n \subset X which can be used to approximate the elements of F. The best possible error we can achieve for such an approximation is given by the Kolmogorov width d_n(F)_X. However, finding the space which gives this performance is typically numerically intractable. Recently, a new greedy strategy for obtaining good spaces was given in the context of the reduced basis method for solving a parametric family of PDEs. The performance of this greedy algorithm was initially analyzed in A. Buffa, Y. Maday, A.T. Patera, C. Prud'homme, and G. Turinici, "A Priori convergence of the greedy algorithm for the parameterized reduced basis", M2AN Math. Model. Numer. Anal., 46(2012), 595-603 in the case X = H is a Hilbert space. The results there were significantly improved on in P. Binev, A. Cohen, W. Dahmen, R. DeVore, G. Petrova, and P. Wojtaszczyk, "Convergence rates for greedy algorithms in reduced bases Methods", SIAM J. Math. Anal., 43 (2011), 1457-1472. The purpose of the present paper is to give a new analysis of the performance of such greedy algorithms. Our analysis not only gives improved results for the Hilbert space case but can also be applied to the same greedy procedure in general Banach spaces.

Functional Analysis
Sparse approximation by greedy algorithms

Vladimir Temlyakov

DOI: https://doi.org/10.48550/arXiv.1511.01846

2015-11-06

Abstract:It is a survey on recent results in constructive sparse approximation. Three directions are discussed here: (1) Lebesgue-type inequalities for greedy algorithms with respect to a special class of dictionaries, (2) constructive sparse approximation with respect to the trigonometric system, (3) sparse approximation with respect to dictionaries with tensor product structure. In all three cases constructive ways are provided for sparse approximation. The technique used is based on fundamental results from the theory of greedy approximation. In particular, results in the direction (1) are based on deep methods developed recently in compressed sensing. We present some of these results with detailed proofs.

Numerical Analysis,Machine Learning
Greedy expansions in convex optimization

V.N. Temlyakov

DOI: https://doi.org/10.48550/arXiv.1206.0393

2012-06-03

Abstract:This paper is a follow up to the previous author's paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there three the most popular in nonlinear approximation in Banach spaces greedy algorithms -- Weak Chebyshev Greedy Algorithm, Weak Greedy Algorithm with Free Relaxation and Weak Relaxed Greedy Algorithm -- for solving convex optimization problems. We continue to study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. In this paper we concentrate on greedy algorithms that provide expansions, which means that the approximant at the $m$th iteration is equal to the sum of the approximant from the previous iteration ($(m-1)$th iteration) and one element from the dictionary with an appropriate coefficient. The problem of greedy expansions of elements of a Banach space is well studied in nonlinear approximation theory. At a first glance the setting of a problem of expansion of a given element and the setting of the problem of expansion in an optimization problem are very different. However, it turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy expansions technique can be adjusted for finding a sparse solution of an optimization problem given by an expansion with respect to a given dictionary.

Machine Learning,Optimization and Control
On greedy approximation in complex Banach spaces

A. Gasnikov,V. Temlyakov

2024-09-29

Abstract:The general theory of greedy approximation with respect to arbitrary dictionaries is well developed in the case of real Banach spaces. Recently, some of results proved for the Weak Chebyshev Greedy Algorithm (WCGA) in the case of real Banach spaces were extended to the case of complex Banach spaces. In this paper we extend some of known in the real case results for other than WCGA greedy algorithms to the case of complex Banach spaces.

Functional Analysis,Numerical Analysis

Comparison of Purely Greedy and Orthogonal Greedy Algorithm

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Projection Greedy Algorithm

On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

On Greedy Algorithms with bounded cumulative coherence

Biorthogonal Greedy Algorithms in convex optimization

Super greedy type algorithms

Greedy algorithms: a review and open problems

Greedy expansions with prescribed coefficients in Hilbert spaces for special classes of dictionaries

A unified way of analyzing some greedy algorithms

Entropy-based convergence rates of greedy algorithms

On the rate of convergence of greedy algorithms

Greed is Good: Algorithmic Results for Sparse Approximation

Greedy Approximation in Convex Optimization

Efficient and provably convergent randomized greedy algorithms for neural network optimization

Greedy Algorithms for Reduced Bases in Banach Spaces

Sparse approximation by greedy algorithms

Greedy expansions in convex optimization

On greedy approximation in complex Banach spaces