On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.1003.5349

2010-03-28

Numerical Analysis

Abstract:We show that Orthogonal Greedy Algorithms (Orthogonal Matching Pursuit) provides almost optimal approximation on the first [1/(20M)] steps for M-coherent dictionaries

What problem does this paper attempt to address?

On Greedy Algorithms with bounded cumulative coherence

Eugene Livshitz

DOI: https://doi.org/10.48550/arXiv.0911.1500

2009-11-08

Abstract:We discuss the upper and lower estimates for the rate of convergence of Pure and Orthogonal Greedy Algorithms for dictionary with bounded cumulative coherence.

Numerical Analysis
Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Jonathan W. Siegel,Jinchao Xu

DOI: https://doi.org/10.1109/tit.2022.3147984

IF: 2.5

2022-05-01

IEEE Transactions on Information Theory

Abstract:We analyze the orthogonal greedy algorithm when applied to dictionaries whose convex hull has small entropy. We show that if the metric entropy of the convex hull of decays at a rate of for , then the orthogonal greedy algorithm converges at the same rate on the variation space of . This improves upon the well-known convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of .

computer science, information systems,engineering, electrical & electronic
Greed is Good: Algorithmic Results for Sparse Approximation

J.A. Tropp

DOI: https://doi.org/10.1109/tit.2004.834793

IF: 2.5

2004-10-01

IEEE Transactions on Information Theory

Abstract:This article presents new results on using a greedy algorithm, orthogonal matching pursuit (OMP), to solve the sparse approximation problem over redundant dictionaries. It provides a sufficient condition under which both OMP and Donoho's basis pursuit (BP) paradigm can recover the optimal representation of an exactly sparse signal. It leverages this theory to show that both OMP and BP succeed for every sparse input signal from a wide class of dictionaries. These quasi-incoherent dictionaries offer a natural generalization of incoherent dictionaries, and the cumulative coherence function is introduced to quantify the level of incoherence. This analysis unifies all the recent results on BP and extends them to OMP. Furthermore, the paper develops a sufficient condition under which OMP can identify atoms from an optimal approximation of a nonsparse signal. From there, it argues that OMP is an approximation algorithm for the sparse problem over a quasi-incoherent dictionary. That is, for every input signal, OMP calculates a sparse approximant whose error is only a small factor worse than the minimal error that can be attained with the same number of terms.

computer science, information systems,engineering, electrical & electronic
Gradient-based Dictionary Optimization for Compressive Spectral Imaging

Chenning Tao,Peng Sun,Siqi Liu,Chang Wang,Jinlei Zhang,Zhanghao Ding,Zhenrong Zheng

DOI: https://doi.org/10.1117/12.2573307

2020-01-01

Abstract:In compressive spectral imaging, three-dimensional spatio-spectral data cubes are recovered from two-dimensional projections. The quality of the compressive-sensing-based reconstruction is dependent on the coherence of the sensing matrix, which is determined by the system projection and the sparse prior. Studies on the optimization of the system projection, which mainly deals with the coded aperture, successfully decreases the coherence of the sensing matrix and improves the reconstruction quality. However, the optimization of the sparse prior considering the relationship between the system projection and the sparse prior remains a challenge. In this paper, we propose a gradient-descent-based sparse prior optimization algorithm for the coherence minimization of the sensing matrix in compressive spectral imaging. The Frobenius norm coherence is introduced as the cost function for the optimization, and the overcomplete dictionary is chosen as the sparse prior to solve the optimal sparse representation in the reconstruction as it provides higher degree of freedom for optimization compared to common orthogonal bases. The optimized dictionary effectively decreases the coherence of the sensing matrix from 0.880 to 0.604 and significantly improves the quantitative image quality metrics of the reconstructed hyperspectral images with the corresponding peak signal-to-noise ratio (PSNR) increased by 9 dB, the structural similarity (SSIM) above 0.98, and the spectrum angular mapper (SAM) below 0.1. Furthermore, the requirement of the sampling snapshots is reduced, which is shown by similar image quality metrics between the reconstructed hyperspectral images of only 1 snapshot with the optimized dictionary and of more than 5 snapshots with the non-optimized dictionary.
On Sparse Recovery Algorithms in Unions of Orthonormal Bases

Yi Shen,Chenyun Yu,Yuan Shen,Song Li

DOI: https://doi.org/10.1016/j.jat.2023.105886

IF: 0.993

2023-01-01

Journal of Approximation Theory

Abstract:We are concerned with the problem of recovering sparse vectors exactly by two commonly used methods, orthogonal matching pursuit (OMP) and basis pursuit (BP). Suppose that overcomplete dictionaries are unions of several orthonormal bases. Gribonval and Nielsen have provided a sufficient mutual coherence condition for BP to recover every sparse vector exactly. Then Tropp proved that it is a sharp sufficient condition for OMP by existence proofs. We establish here the sharpness of Gribonval and Nielsen’s sufficient condition for both OMP and BP by constructing several families of counterexamples. Our main tools are Hadamard matrices and mutually unbiased bases from quantum information theory.
Comparison of Purely Greedy and Orthogonal Greedy Algorithm

Vishnevetskiy, K. S.

DOI: https://doi.org/10.1134/s0001434624010048

2024-04-23

Mathematical Notes

Abstract:Conditions for a dictionary in a Hilbert space are obtained that are necessary or sufficient for the coincidence of purely greedy and orthogonal greedy algorithms with respect to this dictionary.

mathematics
New coherence and rip analysis for weak orthogonal matching pursuit

Mingrui Yang,Frank de Hoog

DOI: https://doi.org/10.1109/ssp.2014.6884654

2014-06-01

Abstract:In this paper we define a new coherence index, named the global 2-coherence, of a given dictionary and study its relationship with the conventional mutual coherence and the restricted isometry constant. By exploring this relationship, we obtain more general results on sparse signal reconstruction using greedy algorithms in the compressive sensing (CS) framework. In particular, we obtain an improved bound over the best known results on the restricted isometry constant for successful recovery of sparse signals using orthogonal matching pursuit (OMP).
Efficient coordinate-descent for orthogonal matrices through Givens rotations

Uri Shalit,Gal Chechik

DOI: https://doi.org/10.48550/arXiv.1312.0624

IF: 5.414

2013-12-02

Machine Learning

Abstract:Optimizing over the set of orthogonal matrices is a central component in problems like sparse-PCA or tensor decomposition. Unfortunately, such optimization is hard since simple operations on orthogonal matrices easily break orthogonality, and correcting orthogonality usually costs a large amount of computation. Here we propose a framework for optimizing orthogonal matrices, that is the parallel of coordinate-descent in Euclidean spaces. It is based on {\em Givens-rotations}, a fast-to-compute operation that affects a small number of entries in the learned matrix, and preserves orthogonality. We show two applications of this approach: an algorithm for tensor decomposition that is used in learning mixture models, and an algorithm for sparse-PCA. We study the parameter regime where a Givens rotation approach converges faster and achieves a superior model on a genome-wide brain-wide mRNA expression dataset.
Super greedy type algorithms

Entao Liu,Vladimir N. Temlyakov

DOI: https://doi.org/10.1007/s10444-011-9220-5

2011-09-30

Advances in Computational Mathematics

Abstract:We study greedy-type algorithms such that at a greedy step we pick several dictionary elements contrary to a single dictionary element in standard greedy algorithms. We call such greedy algorithms super greedy type algorithms. The super greedy type algorithms are computationally simpler than their analogues from the standard greedy algorithms. In this article, we propose the Weak Super Greedy Algorithm (WSGA) and the Weak Orthogonal Super Greedy Algorithm with Thresholding (WOSGAT). Their performance (rate of convergence) are studied as well under M-coherent dictionaries.

mathematics, applied
Projection Greedy Algorithm

P. A. Borodin,S. V. Konyagin

DOI: https://doi.org/10.1134/S0001434621070026

2021-08-24

Mathematical Notes

Abstract:We introduce and study a new type of greedy algorithm, namely, projection greedy algorithms with respect to a given dictionary in a Hilbert space. We prove that these algorithms converge and estimate the rate of convergence for initial elements from the convex hull of the dictionary. Several specific examples of dictionaries are used to compare the introduced algorithms with orthogonal greedy algorithms.

mathematics
Orthogonal Procrustes Analysis for Dictionary Learning in Sparse Linear Representation

Giuliano Grossi,Raffaella Lanzarotti,Jianyi Lin

DOI: https://doi.org/10.1371/journal.pone.0169663

IF: 3.7

2017-01-19

PLoS ONE

Abstract:In the sparse representation model, the design of overcomplete dictionaries plays a key role for the effectiveness and applicability in different domains. Recent research has produced several dictionary learning approaches, being proven that dictionaries learnt by data examples significantly outperform structured ones, e.g. wavelet transforms. In this context, learning consists in adapting the dictionary atoms to a set of training signals in order to promote a sparse representation that minimizes the reconstruction error. Finding the best fitting dictionary remains a very difficult task, leaving the question still open. A well-established heuristic method for tackling this problem is an iterative alternating scheme, adopted for instance in the well-known K-SVD algorithm. Essentially, it consists in repeating two stages; the former promotes sparse coding of the training set and the latter adapts the dictionary to reduce the error. In this paper we present R-SVD, a new method that, while maintaining the alternating scheme, adopts the Orthogonal Procrustes analysis to update the dictionary atoms suitably arranged into groups. Comparative experiments on synthetic data prove the effectiveness of R-SVD with respect to well known dictionary learning algorithms such as K-SVD, ILS-DLA and the online method OSDL. Moreover, experiments on natural data such as ECG compression, EEG sparse representation, and image modeling confirm R-SVD's robustness and wide applicability.
Efficient Implementations for Orthogonal Matching Pursuit

Hufei Zhu,Wen Chen,Yanpeng Wu

DOI: https://doi.org/10.3390/electronics9091507

IF: 2.9

2020-01-01

Electronics

Abstract:Based on the efficient inverse Cholesky factorization, we propose an implementation of OMP (called as version 0, i.e., v0) and its four memory-saving versions (i.e., the proposed v1, v2, v3 and v4). In the simulations, the proposed five versions and the existing OMP implementations have nearly the same numerical errors. Among all the OMP implementations, the proposed v0 needs the least computational complexity, and is the fastest in the simulations for almost all problem sizes. As a tradeoff between computational complexities/time and memory requirements, the proposed v1 seems to be better than all the existing ones when only considering the efficient OMP implementations storing G (i.e., the Gram matrix of the dictionary), the proposed v2 and v3 seem to be better than the only existing one when only considering the efficient implementations not storing G, and the proposed v4 seems to be better than the naive implementation that has the (known) minimum memory requirements. Moreover, all the proposed five versions only include parallelizable matrix-vector products in each iteration, and do not need any back-substitutions that are necessary in some existing efficient implementations (e.g., those utilizing the Cholesky factorization).
A Fast Algorithm for Learning Overcomplete Dictionary for Sparse Representation Based on Proximal Operators

Zhenni Li,Shuxue Ding,Yujie Li

DOI: https://doi.org/10.1162/neco_a_00763

IF: 3.278

2015-01-01

Neural Computation

Abstract:We present a fast, efficient algorithm for learning an overcomplete dictionary for sparse representation of signals. The whole problem is considered as a minimization of the approximation error function with a coherence penalty for the dictionary atoms and with the sparsity regularization of the coefficient matrix. Because the problem is nonconvex and nonsmooth, this minimization problem cannot be solved efficiently by an ordinary optimization method. We propose a decomposition scheme and an alternating optimization that can turn the problem into a set of minimizations of piecewise quadratic and univariate subproblems, each of which is a single variable vector problem, of either one dictionary atom or one coefficient vector. Although the subproblems are still nonsmooth, remarkably they become much simpler so that we can find a closed-form solution by introducing a proximal operator. This leads to an efficient algorithm for sparse representation. To our knowledge, applying the proximal operator to the problem with an incoherence term and obtaining the optimal dictionary atoms in closed form with a proximal operator technique have not previously been studied. The main advantages of the proposed algorithm are that, as suggested by our analysis and simulation study, it has lower computational complexity and a higher convergence rate than state-of-the-art algorithms. In addition, for real applications, it shows good performance and significant reductions in computational time.
Forward Basis Selection for Pursuing Sparse Representations over a Dictionary

Xiao-Tong Yuan,Shuicheng Yan

DOI: https://doi.org/10.1109/tpami.2013.85

IF: 23.6

2013-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:The forward greedy selection algorithm of Frank and Wolfe has recently been applied with success to coordinate-wise sparse learning problems, characterized by a tradeoff between sparsity and accuracy. In this paper, we generalize this method to the setup of pursuing sparse representations over a prefixed dictionary. Our proposed algorithm iteratively selects an atom from the dictionary and minimizes the objective function over the linear combinations of all the selected atoms. The rate of convergence of this greedy selection procedure is analyzed. Furthermore, we extend the algorithm to the setup of learning nonnegative and convex sparse representation over a dictionary. Applications of the proposed algorithms to sparse precision matrix estimation and low-rank subspace segmentation are investigated with efficiency and effectiveness validated on benchmark datasets.
A novel orthogonal matching pursuit algorithm based on reduced-dimension dictionary for airborne MIMO radar

Qiuyue Yin,Jinwang Yi,Jun Tang,Qin Zhu

DOI: https://doi.org/10.1109/ISCEIC53685.2021.00049

2021-01-01

Abstract:In traditional airborne multiple-input multiple-output (MIMO) radar, high correlation of dictionary atoms usually degrades the performance of orthogonal matching pursuit (OMP) algorithm in sparse recovery space-time adaptive processing (SR-STAP). An OMP algorithm based on reduced-dimension dictionary is developed to solve this problem. It divides the dictionary along the clutter ridge and vertical...
Greedy solution of ill-posed problems: error bounds and exact inversion

L Denis,D A Lorenz,D Trede

DOI: https://doi.org/10.1088/0266-5611/25/11/115017

IF: 2.408

2009-11-01

Inverse Problems

Abstract:The orthogonal matching pursuit (OMP) is a greedy algorithm to solve sparse approximation problems. Sufficient conditions for exact recovery are known with and without noise. In this paper we investigate the applicability of the OMP for the solution of ill-posed inverse problems in general, and in particular for two deconvolution examples from mass spectrometry and digital holography, respectively. In sparse approximation problems one often has to deal with the problem of redundancy of a dictionary, i.e. the atoms are not linearly independent. However, one expects them to be approximatively orthogonal and this is quantified by the so-called incoherence. This idea cannot be transferred to ill-posed inverse problems since here the atoms are typically far from orthogonal. The ill-posedness of the operator probably causes the correlation of two distinct atoms to become huge, i.e. that two atoms look much alike. Therefore, one needs conditions which take the structure of the problem into account and work without the concept of coherence. In this paper we develop results for the exact recovery of the support of noisy signals. In the two examples, mass spectrometry and digital holography, we show that our results lead to practically relevant estimates such that one may check a priori if the experimental setup guarantees exact deconvolution with OMP. Especially in the example from digital holography, our analysis may be regarded as a first step to calculate the resolution power of droplet holography.

mathematics, applied,physics, mathematical
Efficient Implementations of Orthogonal Matching Pursuit Based on Inverse Cholesky Factorization

Hufei Zhu,Ganghua Yang,Wen Chen

DOI: https://doi.org/10.1109/VTCFall.2013.6692175

2013-01-01

Abstract:Based on the recently proposed efficient inverse Cholesky factorization, we propose three efficient implementations of Orthogonal Matching Pursuit (OMP), and compare them with the existing implementations of OMP by theoretical and empirical analysis. The proposed implementation 1 theoretically needs the least computational complexity, and is the fastest in the simulations for almost all problem sizes. Among the implementations that store the Gram matrix of the dictionary, the proposed implementation 1 needs the least memories in the k-th iteration (k > 1). As the memory-saving variants of the proposed implementation 1, the proposed implementations 2 and 3 save the memories for the Gram matrix at the expense of higher computational complexity, and in the simulations they are faster than the existing implementations for most problem sizes. With respect to the existing efficient implementation that does not store the Gram matrix, the proposed implementation 2 needs less computational complexity and a little more memories, while the proposed implementation 3 needs the same complexity and less memories.
Convergence of alternating minimisation algorithms for dictionary learning

Simon Ruetz,Karin Schnass

2023-05-26

Abstract:In this paper we derive sufficient conditions for the convergence of two popular alternating minimisation algorithms for dictionary learning - the Method of Optimal Directions (MOD) and Online Dictionary Learning (ODL), which can also be thought of as approximative K-SVD. We show that given a well-behaved initialisation that is either within distance at most $1/\log(K)$ to the generating dictionary or has a special structure ensuring that each element of the initialisation only points to one generating element, both algorithms will converge with geometric convergence rate to the generating dictionary. This is done even for data models with non-uniform distributions on the supports of the sparse coefficients. These allow the appearance frequency of the dictionary elements to vary heavily and thus model real data more closely.

Optimization and Control,Machine Learning
Complete Dictionary Learning via $\ell^4$-Norm Maximization over the Orthogonal Group

Yuexiang Zhai,Zitong Yang,Zhenyu Liao,John Wright,Yi Ma

DOI: https://doi.org/10.48550/arXiv.1906.02435

2021-04-07

Abstract:This paper considers the fundamental problem of learning a complete (orthogonal) dictionary from samples of sparsely generated signals. Most existing methods solve the dictionary (and sparse representations) based on heuristic algorithms, usually without theoretical guarantees for either optimality or complexity. The recent $\ell^1$-minimization based methods do provide such guarantees but the associated algorithms recover the dictionary one column at a time. In this work, we propose a new formulation that maximizes the $\ell^4$-norm over the orthogonal group, to learn the entire dictionary. We prove that under a random data model, with nearly minimum sample complexity, the global optima of the $\ell^4$ norm are very close to signed permutations of the ground truth. Inspired by this observation, we give a conceptually simple and yet effective algorithm based on "matching, stretching, and projection" (MSP). The algorithm provably converges locally at a superlinear (cubic) rate and cost per iteration is merely an SVD. In addition to strong theoretical guarantees, experiments show that the new algorithm is significantly more efficient and effective than existing methods, including KSVD and $\ell^1$-based methods. Preliminary experimental results on mixed real imagery data clearly demonstrate advantages of so learned dictionary over classic PCA bases.

Machine Learning,Signal Processing,Computation
Learning and Approximation Capabilities of Orthogonal Super Greedy Algorithm

Jian Fang,Shaobo Lin,Zongben Xu

DOI: https://doi.org/10.1016/j.knosys.2015.12.011

IF: 8.139

2016-01-01

Knowledge-Based Systems

Abstract:We consider the approximation capability of orthogonal super greedy algorithms (OSGA) and its applications in supervised learning. OSGA focuses on selecting more than one atoms in each iteration, which, of course, reduces the computational burden when compared with the conventional orthogonal greedy algorithm (OGA). We prove that even for function classes that are not the convex hull of the dictionary, OSGA does not degrade the approximation capability of OGA, provided the dictionary is incoherent. Based on this, we deduce tight generalization error bounds for OSGA learning. Our results show that in the realm of supervised learning, OSGA provides a possibility to further reduce the computational burden of OGA on the premise of maintaining its prominent generalization capability.

On the optimality of Orthogonal Greedy Algorithm for M-coherent dictionaries

On Greedy Algorithms with bounded cumulative coherence

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

Greed is Good: Algorithmic Results for Sparse Approximation

Gradient-based Dictionary Optimization for Compressive Spectral Imaging

On Sparse Recovery Algorithms in Unions of Orthonormal Bases

Comparison of Purely Greedy and Orthogonal Greedy Algorithm

New coherence and rip analysis for weak orthogonal matching pursuit

Efficient coordinate-descent for orthogonal matrices through Givens rotations

Super greedy type algorithms

Projection Greedy Algorithm

Orthogonal Procrustes Analysis for Dictionary Learning in Sparse Linear Representation

Efficient Implementations for Orthogonal Matching Pursuit

A Fast Algorithm for Learning Overcomplete Dictionary for Sparse Representation Based on Proximal Operators

Forward Basis Selection for Pursuing Sparse Representations over a Dictionary

A novel orthogonal matching pursuit algorithm based on reduced-dimension dictionary for airborne MIMO radar

Greedy solution of ill-posed problems: error bounds and exact inversion

Efficient Implementations of Orthogonal Matching Pursuit Based on Inverse Cholesky Factorization

Convergence of alternating minimisation algorithms for dictionary learning

Complete Dictionary Learning via $\ell^4$-Norm Maximization over the Orthogonal Group

Learning and Approximation Capabilities of Orthogonal Super Greedy Algorithm