On Greedy Algorithms with bounded cumulative coherence

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.0911.1500

2009-11-08

Abstract:We discuss the upper and lower estimates for the rate of convergence of Pure and Orthogonal Greedy Algorithms for dictionary with bounded cumulative coherence.

Numerical Analysis

What problem does this paper attempt to address?

On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.1003.5349

2010-03-28

Numerical Analysis

Abstract:We show that Orthogonal Greedy Algorithms (Orthogonal Matching Pursuit) provides almost optimal approximation on the first [1/(20M)] steps for M-coherent dictionaries
Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Jonathan W. Siegel,Jinchao Xu

DOI: https://doi.org/10.1109/tit.2022.3147984

IF: 2.5

2022-05-01

IEEE Transactions on Information Theory

Abstract:We analyze the orthogonal greedy algorithm when applied to dictionaries whose convex hull has small entropy. We show that if the metric entropy of the convex hull of decays at a rate of for , then the orthogonal greedy algorithm converges at the same rate on the variation space of . This improves upon the well-known convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of .

computer science, information systems,engineering, electrical & electronic
On the rate of convergence of greedy algorithms

V.N. Temlyakov

2023-04-13

Abstract:We prove some results on the rate of convergence of greedy algorithms, which provide expansions. We consider both the case of Hilbert spaces and the more general case of Banach spaces. The new ingredient of the paper is that we bound the error of approximation by the product of both norms -- the norm of $f$ and the $A_1$-norm of $f$. Typically, only the $A_1$-norm of $f$ is used. In particular, we establish that some greedy algorithms (Pure Greedy Algorithm (PGA) and its generalizations) are as good as the Orthogonal Greedy Algorithm (OGA) in this new sense of the rate of convergence, while it is known that the PGA is much worth than the OGA in the standard sense.

Numerical Analysis,Functional Analysis
Super greedy type algorithms

Entao Liu,Vladimir N. Temlyakov

DOI: https://doi.org/10.1007/s10444-011-9220-5

2011-09-30

Advances in Computational Mathematics

Abstract:We study greedy-type algorithms such that at a greedy step we pick several dictionary elements contrary to a single dictionary element in standard greedy algorithms. We call such greedy algorithms super greedy type algorithms. The super greedy type algorithms are computationally simpler than their analogues from the standard greedy algorithms. In this article, we propose the Weak Super Greedy Algorithm (WSGA) and the Weak Orthogonal Super Greedy Algorithm with Thresholding (WOSGAT). Their performance (rate of convergence) are studied as well under M-coherent dictionaries.

mathematics, applied
Comparison of Purely Greedy and Orthogonal Greedy Algorithm

Vishnevetskiy, K. S.

DOI: https://doi.org/10.1134/s0001434624010048

2024-04-23

Mathematical Notes

Abstract:Conditions for a dictionary in a Hilbert space are obtained that are necessary or sufficient for the coincidence of purely greedy and orthogonal greedy algorithms with respect to this dictionary.

mathematics
Entropy-based convergence rates of greedy algorithms

Yuwen Li,Jonathan W. Siegel

DOI: https://doi.org/10.1142/s0218202524500143

2024-02-20

Mathematical Models and Methods in Applied Sciences

Abstract:Mathematical Models and Methods in Applied Sciences, Ahead of Print. We present convergence estimates of two types of greedy algorithms in terms of the entropy numbers of underlying compact sets. In the first part, we measure the error of a standard greedy reduced basis method for parametric PDEs by the entropy numbers of the solution manifold in Banach spaces. This contrasts with the classical analysis based on the Kolmogorov [math]-widths and enables us to obtain direct comparisons between the algorithm error and the entropy numbers, where the multiplicative constants are explicit and simple. The entropy-based convergence estimate is sharp and improves upon the classical width-based analysis of reduced basis methods for elliptic model problems. In the second part, we derive a novel and simple convergence analysis of the classical orthogonal greedy algorithm for nonlinear dictionary approximation using the entropy numbers of the symmetric convex hull of the dictionary. This also improves upon existing results by giving a direct comparison between the algorithm error and the entropy numbers.

mathematics, applied
Projection Greedy Algorithm

P. A. Borodin,S. V. Konyagin

DOI: https://doi.org/10.1134/S0001434621070026

2021-08-24

Mathematical Notes

Abstract:We introduce and study a new type of greedy algorithm, namely, projection greedy algorithms with respect to a given dictionary in a Hilbert space. We prove that these algorithms converge and estimate the rate of convergence for initial elements from the convex hull of the dictionary. Several specific examples of dictionaries are used to compare the introduced algorithms with orthogonal greedy algorithms.

mathematics
Rate of convergence of Thresholding Greedy Algorithms

Vladimir Nikolaevich Temlyakov

DOI: https://doi.org/10.4213/sm9926e

2024-06-22

Sbornik Mathematics

Abstract:The rate of convergence of the classical Thresholding Greedy Algorithm with respect to some bases is studied. We bound the error of approximation by the product of two norms, the norm of and the -norm of . We obtain some results for greedy bases, unconditional bases and quasi-greedy bases. In particular, we prove that our bounds for the trigonometric basis and Haar basis are optimal. Bibliography: 16 titles.

mathematics
Greedy expansions with prescribed coefficients in Hilbert spaces for special classes of dictionaries

Alessandro Oliaro,Luca Tomatis,Albert R. Valiullin,Artur R. Valiullin

2023-07-05

Abstract:Greedy expansions with prescribed coefficients have been introduced by V. N. Temlyakov in the frame of Banach spaces. The idea is to choose a sequence of fixed (real) coefficients $\{c_n\}_{n=1}^\infty$ and a fixed set of elements (dictionary) of the Banach space; then, under suitable conditions on the coefficients and the dictionary, it is possible to expand all the elements of the Banach space in series that contain only the fixed coefficients and the elements of the dictionary. In Hilbert spaces the convergence of greedy algorithm with prescribed coefficients is characterized, in the sense that there are necessary and sufficient conditions on the coefficients in order that the algorithm is convergent for all the dictionaries. This paper is concerned with the question if such conditions can be weakened for particular classes of spaces or dictionaries; we prove that this is the case for finite dimensional spaces, and for some classes of dictionaries related to orthonormal sequences in infinite dimensional spaces.

Functional Analysis
Greedy algorithms: a review and open problems

Andrea García

2024-08-16

Abstract:Greedy algorithms are a fundamental category of algorithms in mathematics and computer science, characterized by their iterative, locally optimal decision-making approach, which aims to find global optima. In this review, we will discuss two greedy algorithms. First, we will talk about the so-called Relaxed Greedy Algorithm in the context of dictionaries in Hilbert spaces analyzing the optimality of definition of this algorithm and, next, we give a general overview of the Thresholding Greedy Algorithm and the Chebyshev Thresholding Greedy Algorithm with regard to bases in p-Banach spaces with $0 < p \leq 1$. In both cases, we pose some questions for future research.

Functional Analysis
Convergence and rate of convergence of some greedy algorithms in convex optimization

Vladimir Temlyakov

DOI: https://doi.org/10.48550/arXiv.1412.3297

2014-12-10

Abstract:The paper gives a systematic study of the approximate versions of three greedy-type algorithms that are widely used in convex optimization. By approximate version we mean the one where some of evaluations are made with an error. Importance of such versions of greedy-type algorithms in convex optimization and in approximation theory was emphasized in previous literature.

Machine Learning,Numerical Analysis
New coherence and rip analysis for weak orthogonal matching pursuit

Mingrui Yang,Frank de Hoog

DOI: https://doi.org/10.1109/ssp.2014.6884654

2014-06-01

Abstract:In this paper we define a new coherence index, named the global 2-coherence, of a given dictionary and study its relationship with the conventional mutual coherence and the restricted isometry constant. By exploring this relationship, we obtain more general results on sparse signal reconstruction using greedy algorithms in the compressive sensing (CS) framework. In particular, we obtain an improved bound over the best known results on the restricted isometry constant for successful recovery of sparse signals using orthogonal matching pursuit (OMP).
Biorthogonal Greedy Algorithms in convex optimization

A.V. Dereventsov,V.N. Temlyakov

DOI: https://doi.org/10.1016/j.acha.2022.05.001

IF: 2.974

2022-09-01

Applied and Computational Harmonic Analysis

Abstract:The study of greedy approximation in the context of convex optimization is becoming a promising research direction as greedy algorithms are actively being employed to construct sparse minimizers for convex functions with respect to given sets of elements. In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms for the minimization of convex functions on Banach spaces. Specifically, we define the class of Weak Biorthogonal Greedy Algorithms for convex optimization that contains a wide range of greedy algorithms. We analyze the introduced class of algorithms and establish the properties of convergence, rate of convergence, and numerical stability, which is understood in the sense that the steps of the algorithm are allowed to be performed not precisely but with controlled computational inaccuracies. We show that the following well-known algorithms for convex optimization — the Weak Chebyshev Greedy Algorithm (co) and the Weak Greedy Algorithm with Free Relaxation (co) — belong to this class, and introduce a new algorithm — the Rescaled Weak Relaxed Greedy Algorithm (co). Presented numerical experiments demonstrate the practical performance of the aforementioned greedy algorithms in the setting of convex minimization as compared to optimization with regularization, which is the conventional approach of constructing sparse minimizers.

mathematics, applied
Greed is Good: Algorithmic Results for Sparse Approximation

J.A. Tropp

DOI: https://doi.org/10.1109/tit.2004.834793

IF: 2.5

2004-10-01

IEEE Transactions on Information Theory

Abstract:This article presents new results on using a greedy algorithm, orthogonal matching pursuit (OMP), to solve the sparse approximation problem over redundant dictionaries. It provides a sufficient condition under which both OMP and Donoho's basis pursuit (BP) paradigm can recover the optimal representation of an exactly sparse signal. It leverages this theory to show that both OMP and BP succeed for every sparse input signal from a wide class of dictionaries. These quasi-incoherent dictionaries offer a natural generalization of incoherent dictionaries, and the cumulative coherence function is introduced to quantify the level of incoherence. This analysis unifies all the recent results on BP and extends them to OMP. Furthermore, the paper develops a sufficient condition under which OMP can identify atoms from an optimal approximation of a nonsparse signal. From there, it argues that OMP is an approximation algorithm for the sparse problem over a quasi-incoherent dictionary. That is, for every input signal, OMP calculates a sparse approximant whose error is only a small factor worse than the minimal error that can be attained with the same number of terms.

computer science, information systems,engineering, electrical & electronic
On the Convergence of Bound Optimization Algorithms

Ruslan R Salakhutdinov,Sam T Roweis,Zoubin Ghahramani

DOI: https://doi.org/10.48550/arXiv.1212.2490

2012-10-19

Abstract:Many practitioners who use the EM algorithm complain that it is sometimes slow. When does this happen, and what can be done about it? In this paper, we study the general class of bound optimization algorithms - including Expectation-Maximization, Iterative Scaling and CCCP - and their relationship to direct optimization algorithms such as gradient-based methods for parameter learning. We derive a general relationship between the updates performed by bound optimization methods and those of gradient and second-order methods and identify analytic conditions under which bound optimization algorithms exhibit quasi-Newton behavior, and conditions under which they possess poor, first-order convergence. Based on this analysis, we consider several specific algorithms, interpret and analyze their convergence properties and provide some recipes for preprocessing input to these algorithms to yield faster convergence behavior. We report empirical results supporting our analysis and showing that simple data preprocessing can result in dramatically improved performance of bound optimizers in practice.

Machine Learning
Asynchronous Coordinate Descent under More Realistic Assumptions

Tao Sun,Robert Hannah,Wotao Yin

DOI: https://doi.org/10.48550/arXiv.1705.08494

2017-07-19

Abstract:Asynchronous-parallel algorithms have the potential to vastly speed up algorithms by eliminating costly synchronization. However, our understanding to these algorithms is limited because the current convergence of asynchronous (block) coordinate descent algorithms are based on somewhat unrealistic assumptions. In particular, the age of the shared optimization variables being used to update a block is assumed to be independent of the block being updated. Also, it is assumed that the updates are applied to randomly chosen blocks. In this paper, we argue that these assumptions either fail to hold or will imply less efficient implementations. We then prove the convergence of asynchronous-parallel block coordinate descent under more realistic assumptions, in particular, always without the independence assumption. The analysis permits both the deterministic (essentially) cyclic and random rules for block choices. Because a bound on the asynchronous delays may or may not be available, we establish convergence for both bounded delays and unbounded delays. The analysis also covers nonconvex, weakly convex, and strongly convex functions. We construct Lyapunov functions that directly model both objective progress and delays, so delays are not treated errors or noise. A continuous-time ODE is provided to explain the construction at a high level.

Optimization and Control,Numerical Analysis
Sparse approximation by greedy algorithms

Vladimir Temlyakov

DOI: https://doi.org/10.48550/arXiv.1511.01846

2015-11-06

Abstract:It is a survey on recent results in constructive sparse approximation. Three directions are discussed here: (1) Lebesgue-type inequalities for greedy algorithms with respect to a special class of dictionaries, (2) constructive sparse approximation with respect to the trigonometric system, (3) sparse approximation with respect to dictionaries with tensor product structure. In all three cases constructive ways are provided for sparse approximation. The technique used is based on fundamental results from the theory of greedy approximation. In particular, results in the direction (1) are based on deep methods developed recently in compressed sensing. We present some of these results with detailed proofs.

Numerical Analysis,Machine Learning

On Greedy Algorithms with bounded cumulative coherence

On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

On the rate of convergence of greedy algorithms

Super greedy type algorithms

Comparison of Purely Greedy and Orthogonal Greedy Algorithm

Entropy-based convergence rates of greedy algorithms

Projection Greedy Algorithm

Rate of convergence of Thresholding Greedy Algorithms

Greedy expansions with prescribed coefficients in Hilbert spaces for special classes of dictionaries

Greedy algorithms: a review and open problems

Convergence and rate of convergence of some greedy algorithms in convex optimization

New coherence and rip analysis for weak orthogonal matching pursuit

Biorthogonal Greedy Algorithms in convex optimization

Greed is Good: Algorithmic Results for Sparse Approximation

On the Convergence of Bound Optimization Algorithms

Asynchronous Coordinate Descent under More Realistic Assumptions

Sparse approximation by greedy algorithms