Rescaled Pure Greedy Algorithm for Convex Optimization

Zheming Gao,Guergana Petrova

DOI: https://doi.org/10.48550/arXiv.1505.03606

2015-05-14

Abstract:We suggest a new greedy strategy for convex optimization in Banach spaces and prove its convergent rates under a suitable behavior of the modulus of uniform smoothness of the objective function.

Optimization and Control

What problem does this paper attempt to address?

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Yu Chen,Xie Xiaoliang

2010-01-01

Journal of applied mathematics & informatics

Abstract:In this paper, A FSQP algorithm for degenerate inequality constraints optimization problems is proposed. At each iteration of the proposed algorithm, a feasible direction of descent is obtained by solving a quadratic programming subproblem. To overcome the Maratos effect, a higher-order correction direction is obtained by solving another quadratic programming subproblem. The algorithm is proved to be globally convergent and superlinearly convergent under some mild conditions. Finally, some preliminary numerical results are reported.
Biorthogonal Greedy Algorithms in convex optimization

A.V. Dereventsov,V.N. Temlyakov

DOI: https://doi.org/10.1016/j.acha.2022.05.001

IF: 2.974

2022-09-01

Applied and Computational Harmonic Analysis

Abstract:The study of greedy approximation in the context of convex optimization is becoming a promising research direction as greedy algorithms are actively being employed to construct sparse minimizers for convex functions with respect to given sets of elements. In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms for the minimization of convex functions on Banach spaces. Specifically, we define the class of Weak Biorthogonal Greedy Algorithms for convex optimization that contains a wide range of greedy algorithms. We analyze the introduced class of algorithms and establish the properties of convergence, rate of convergence, and numerical stability, which is understood in the sense that the steps of the algorithm are allowed to be performed not precisely but with controlled computational inaccuracies. We show that the following well-known algorithms for convex optimization — the Weak Chebyshev Greedy Algorithm (co) and the Weak Greedy Algorithm with Free Relaxation (co) — belong to this class, and introduce a new algorithm — the Rescaled Weak Relaxed Greedy Algorithm (co). Presented numerical experiments demonstrate the practical performance of the aforementioned greedy algorithms in the setting of convex minimization as compared to optimization with regularization, which is the conventional approach of constructing sparse minimizers.

mathematics, applied
Convergence and rate of convergence of some greedy algorithms in convex optimization

Vladimir Temlyakov

DOI: https://doi.org/10.48550/arXiv.1412.3297

2014-12-10

Abstract:The paper gives a systematic study of the approximate versions of three greedy-type algorithms that are widely used in convex optimization. By approximate version we mean the one where some of evaluations are made with an error. Importance of such versions of greedy-type algorithms in convex optimization and in approximation theory was emphasized in previous literature.

Machine Learning,Numerical Analysis
On the rate of convergence of greedy algorithms

V.N. Temlyakov

2023-04-13

Abstract:We prove some results on the rate of convergence of greedy algorithms, which provide expansions. We consider both the case of Hilbert spaces and the more general case of Banach spaces. The new ingredient of the paper is that we bound the error of approximation by the product of both norms -- the norm of $f$ and the $A_1$-norm of $f$. Typically, only the $A_1$-norm of $f$ is used. In particular, we establish that some greedy algorithms (Pure Greedy Algorithm (PGA) and its generalizations) are as good as the Orthogonal Greedy Algorithm (OGA) in this new sense of the rate of convergence, while it is known that the PGA is much worth than the OGA in the standard sense.

Numerical Analysis,Functional Analysis
A Randomized Nonlinear Rescaling Method in Large-Scale Constrained Convex Optimization

Wei Bo,Haskell William B.,Zhao Sixiang

2020-01-01

Abstract: We propose a new randomized algorithm for solving convex optimization problems that have a large number of constraints (with high probability). Existing methods like interior-point or Newton-type algorithms are hard to apply to such problems because they have expensive computation and storage requirements for Hessians and matrix inversions. Our algorithm is based on nonlinear rescaling (NLR), which is a primal-dual-type algorithm by Griva and Polyak {[{Math. Program., 106(2):237-259, 2006}]}. NLR introduces an equivalent problem through a transformation of the constraint functions, minimizes the corresponding augmented Lagrangian for given dual variables, and then uses this minimizer to update the dual variables for the next iteration. The primal update at each iteration is the solution of an unconstrained finite sum minimization problem where the terms are weighted by the current dual variables. We use randomized first-order algorithms to do these primal updates, for which they are especially well suited. In particular, we use the scaled dual variables as the sampling distribution for each primal update, and we show that this distribution is the optimal one among all probability distributions. We conclude by demonstrating the favorable numerical performance of our algorithm.
Greedy Approximation in Convex Optimization

V. N. Temlyakov

DOI: https://doi.org/10.1007/s00365-014-9272-0

2015-01-21

Constructive Approximation

Abstract:We study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of a few elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. The problem of approximation of a given element of a Banach space by linear combinations of elements from a given system (dictionary) is well studied in nonlinear approximation theory. At first glance, the settings of approximation and optimization problems are very different. In the approximation problem, an element is given and our task is to find a sparse approximation of it. In optimization theory, an energy function is given and we should find an approximate sparse solution to the minimization problem. It turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular the greedy approximation technique, can be adjusted for finding a sparse solution of an optimization problem.

mathematics
A unified way of analyzing some greedy algorithms

Anton Dereventsov,Vladimir Temlyakov

DOI: https://doi.org/10.1016/j.jfa.2019.108286

2019-06-05

Abstract:In this paper we propose a unified way of analyzing a certain kind of greedy-type algorithms in Banach spaces. We define a class of the Weak Biorthogonal Greedy Algorithms that contains a wide range of greedy algorithms. In particular, we show that the following well-known algorithms --- the Weak Chebyshev Greedy Algorithm and the Weak Greedy Algorithm with Free Relaxation --- belong to this class. We investigate the properties of convergence, rate of convergence, and numerical stability of the Weak Biorthogonal Greedy Algorithms. Numerical stability is understood in the sense that the steps of the algorithm are allowed to be performed with controlled computational inaccuracies. We carry out a thorough analysis of the connection between the magnitude of those inaccuracies and the convergence properties of the algorithm. To emphasize the advantage of the proposed approach, we introduce here a new greedy algorithm --- the Rescaled Weak Relaxed Greedy Algorithm --- from the above class, and derive the convergence results without analyzing the algorithm explicitly. Additionally, we explain how the proposed approach can be extended to some other types of greedy algorithms.

Numerical Analysis
Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Nam Nguyen,Deanna Needell,Tina Woolf

DOI: https://doi.org/10.1109/tit.2017.2749330

IF: 2.5

2017-11-01

IEEE Transactions on Information Theory

Abstract:Motivated by recent work on stochastic gradient descent methods, we develop two stochastic variants of greedy algorithms for possibly non-convex optimization problems with sparsity constraints. We prove linear convergence1 in expectation to the solution within a specified tolerance. This generalized framework is specialized to the problems of sparse signal recovery in compressed sensing and low-rank matrix recovery, giving methods with provable convergence guarantees that often outperform their deterministic counterparts. We also analyze the settings, where gradients and projections can only be computed approximately, and prove the methods are robust to these approximations. We include many numerical experiments, which align with the theoretical analysis and demonstrate these improvements in several different settings.

computer science, information systems,engineering, electrical & electronic
Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees

Francesco Locatello,Michael Tschannen,Gunnar Rätsch,Martin Jaggi

DOI: https://doi.org/10.48550/arXiv.1705.11041

IF: 5.414

2017-05-31

Machine Learning

Abstract:Greedy optimization methods such as Matching Pursuit (MP) and Frank-Wolfe (FW) algorithms regained popularity in recent years due to their simplicity, effectiveness and theoretical guarantees. MP and FW address optimization over the linear span and the convex hull of a set of atoms, respectively. In this paper, we consider the intermediate case of optimization over the convex cone, parametrized as the conic hull of a generic atom set, leading to the first principled definitions of non-negative MP algorithms for which we give explicit convergence rates and demonstrate excellent empirical performance. In particular, we derive sublinear ($\mathcal{O}(1/t)$) convergence on general smooth and convex objectives, and linear convergence ($\mathcal{O}(e^{-t})$) on strongly convex objectives, in both cases for general sets of atoms. Furthermore, we establish a clear correspondence of our algorithms to known algorithms from the MP and FW literature. Our novel algorithms and analyses target general atom sets and general objective functions, and hence are directly applicable to a large variety of learning settings.
Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

Dachao Lin,Haishan Ye,Zhihua Zhang

DOI: https://doi.org/10.48550/arxiv.2104.08764

2022-01-01

Abstract: Optimization is important in machine learning problems, and quasi-Newton methods have a reputation as the most efficient numerical schemes for smooth unconstrained optimization. In this paper, we consider the explicit superlinear convergence rates of quasi-Newton methods and address two open problems mentioned by Rodomanov and Nesterov. First, we extend Rodomanov and Nesterov's results to random quasi-Newton methods, which include common DFP, BFGS, SR1 methods. Such random methods adopt a random direction for updating the approximate Hessian matrix in each iteration. Second, we focus on the specific quasi-Newton methods: SR1 and BFGS methods. We provide improved versions of greedy and random methods with provable better explicit (local) superlinear convergence rates. Our analysis is closely related to the approximation of a given Hessian matrix, unconstrained quadratic objective, as well as the general strongly convex, smooth, and strongly self-concordant functions.
Entropy-based convergence rates of greedy algorithms

Yuwen Li,Jonathan W. Siegel

DOI: https://doi.org/10.1142/s0218202524500143

2024-02-20

Mathematical Models and Methods in Applied Sciences

Abstract:Mathematical Models and Methods in Applied Sciences, Ahead of Print. We present convergence estimates of two types of greedy algorithms in terms of the entropy numbers of underlying compact sets. In the first part, we measure the error of a standard greedy reduced basis method for parametric PDEs by the entropy numbers of the solution manifold in Banach spaces. This contrasts with the classical analysis based on the Kolmogorov [math]-widths and enables us to obtain direct comparisons between the algorithm error and the entropy numbers, where the multiplicative constants are explicit and simple. The entropy-based convergence estimate is sharp and improves upon the classical width-based analysis of reduced basis methods for elliptic model problems. In the second part, we derive a novel and simple convergence analysis of the classical orthogonal greedy algorithm for nonlinear dictionary approximation using the entropy numbers of the symmetric convex hull of the dictionary. This also improves upon existing results by giving a direct comparison between the algorithm error and the entropy numbers.

mathematics, applied
A Proximal-Gradient Method for Constrained Optimization

Yutong Dai,Xiaoyi Qu,Daniel P. Robinson

2024-04-11

Abstract:We present a new algorithm for solving optimization problems with objective functions that are the sum of a smooth function and a (potentially) nonsmooth regularization function, and nonlinear equality constraints. The algorithm may be viewed as an extension of the well-known proximal-gradient method that is applicable when constraints are not present. To account for nonlinear equality constraints, we combine a decomposition procedure for computing trial steps with an exact merit function for determining trial step acceptance. Under common assumptions, we show that both the proximal parameter and merit function parameter eventually remain fixed, and then prove a worst-case complexity result for the maximum number of iterations before an iterate satisfying approximate first-order optimality conditions for a given tolerance is computed. Our preliminary numerical results indicate that our approach has great promise, especially in terms of returning approximate solutions that are structured (e.g., sparse solutions when a one-norm regularizer is used).

Optimization and Control
Greedy expansions in convex optimization

V.N. Temlyakov

DOI: https://doi.org/10.48550/arXiv.1206.0393

2012-06-03

Abstract:This paper is a follow up to the previous author's paper on convex optimization. In that paper we began the process of adjusting greedy-type algorithms from nonlinear approximation for finding sparse solutions of convex optimization problems. We modified there three the most popular in nonlinear approximation in Banach spaces greedy algorithms -- Weak Chebyshev Greedy Algorithm, Weak Greedy Algorithm with Free Relaxation and Weak Relaxed Greedy Algorithm -- for solving convex optimization problems. We continue to study sparse approximate solutions to convex optimization problems. It is known that in many engineering applications researchers are interested in an approximate solution of an optimization problem as a linear combination of elements from a given system of elements. There is an increasing interest in building such sparse approximate solutions using different greedy-type algorithms. In this paper we concentrate on greedy algorithms that provide expansions, which means that the approximant at the $m$th iteration is equal to the sum of the approximant from the previous iteration ($(m-1)$th iteration) and one element from the dictionary with an appropriate coefficient. The problem of greedy expansions of elements of a Banach space is well studied in nonlinear approximation theory. At a first glance the setting of a problem of expansion of a given element and the setting of the problem of expansion in an optimization problem are very different. However, it turns out that the same technique can be used for solving both problems. We show how the technique developed in nonlinear approximation theory, in particular, the greedy expansions technique can be adjusted for finding a sparse solution of an optimization problem given by an expansion with respect to a given dictionary.

Machine Learning,Optimization and Control
A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Ganzhao Yuan,Li Shen,Wei-Shi Zheng

DOI: https://doi.org/10.48550/arXiv.1706.06493

2017-06-20

Optimization and Control

Abstract:Discrete optimization is a central problem in mathematical optimization with a broad range of applications, among which binary optimization and sparse optimization are two common ones. However, these problems are NP-hard and thus difficult to solve in general. Combinatorial search methods such as branch-and-bound and exhaustive search find the global optimal solution but are confined to small-sized problems, while coordinate descent methods such as coordinate gradient descent are efficient but often suffer from poor local minima. In this paper, we consider a hybrid method that combines the effectiveness of combinatorial search and the efficiency of coordinate descent. Specifically, we consider random strategy or/and greedy strategy to select a subset of coordinates as the working set, and then perform global combinatorial search over the working set based on the original objective function. In addition, we provide some optimality analysis and convergence analysis for the proposed method. Our method finds stronger stationary points than existing methods. Finally, we demonstrate the efficacy of our method on some sparse optimization and binary optimization applications. As a result, our method achieves state-of-the-art performance in terms of accuracy. For example, our method generally outperforms the well-known orthogonal matching pursuit method in sparse optimization. Keywords: Sparsity Optimization, Binary Optimization, Coordinate Descent, Combinatorial Search, Discrete Optimization, Convergence Analysis.
Analyzing and Improving Greedy 2-Coordinate Updates for Equality-Constrained Optimization via Steepest Descent in the 1-Norm

Amrutha Varshini Ramesh,Aaron Mishkin,Mark Schmidt,Yihan Zhou,Jonathan Wilder Lavington,Jennifer She

2023-07-04

Abstract:We consider minimizing a smooth function subject to a summation constraint over its variables. By exploiting a connection between the greedy 2-coordinate update for this problem and equality-constrained steepest descent in the 1-norm, we give a convergence rate for greedy selection under a proximal Polyak-Lojasiewicz assumption that is faster than random selection and independent of the problem dimension $n$. We then consider minimizing with both a summation constraint and bound constraints, as arises in the support vector machine dual problem. Existing greedy rules for this setting either guarantee trivial progress only or require $O(n^2)$ time to compute. We show that bound- and summation-constrained steepest descent in the L1-norm guarantees more progress per iteration than previous rules and can be computed in only $O(n \log n)$ time.

Optimization and Control,Machine Learning
Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence.

Dachao Lin,Haishan Ye,Zhihua Zhang

2021-01-01

Abstract:In this paper, we follow Rodomanov and Nesterov [19]’s work to study quasi-Newton methods. We focus on the common SR1 and BFGS quasi-Newton methods to establish better explicit (local) superlinear convergence rates. First, based on the greedy quasi-Newton update which greedily selects the direction to maximize a certain measure of progress, we improve the convergence rate to a condition-number-free superlinear convergence rate. Second, based on the random quasi-Newton update that selects the direction randomly from a spherically symmetric distribution, we show the same superlinear convergence rate established as above. Our analysis is closely related to the approximation of a given Hessian matrix, unconstrained quadratic objective, as well as the general strongly convex, smooth and strongly self-concordant functions.
Greedy Algorithms for Reduced Bases in Banach Spaces

Ronald DeVore,Guergana Petrova,Przemyslaw Wojtaszczyk

DOI: https://doi.org/10.48550/arXiv.1204.2290

2012-04-11

Abstract:Given a Banach space X and one of its compact sets F, we consider the problem of finding a good n dimensional space X_n \subset X which can be used to approximate the elements of F. The best possible error we can achieve for such an approximation is given by the Kolmogorov width d_n(F)_X. However, finding the space which gives this performance is typically numerically intractable. Recently, a new greedy strategy for obtaining good spaces was given in the context of the reduced basis method for solving a parametric family of PDEs. The performance of this greedy algorithm was initially analyzed in A. Buffa, Y. Maday, A.T. Patera, C. Prud'homme, and G. Turinici, "A Priori convergence of the greedy algorithm for the parameterized reduced basis", M2AN Math. Model. Numer. Anal., 46(2012), 595-603 in the case X = H is a Hilbert space. The results there were significantly improved on in P. Binev, A. Cohen, W. Dahmen, R. DeVore, G. Petrova, and P. Wojtaszczyk, "Convergence rates for greedy algorithms in reduced bases Methods", SIAM J. Math. Anal., 43 (2011), 1457-1472. The purpose of the present paper is to give a new analysis of the performance of such greedy algorithms. Our analysis not only gives improved results for the Hilbert space case but can also be applied to the same greedy procedure in general Banach spaces.

Functional Analysis
Convex and Non-convex Optimization Under Generalized Smoothness

Haochuan Li,Jian Qian,Yi Tian,Alexander Rakhlin,Ali Jadbabaie

2023-11-03

Abstract:Classical analysis of convex and non-convex optimization methods often requires the Lipshitzness of the gradient, which limits the analysis to functions bounded by quadratics. Recent work relaxed this requirement to a non-uniform smoothness condition with the Hessian norm bounded by an affine function of the gradient norm, and proved convergence in the non-convex setting via gradient clipping, assuming bounded noise. In this paper, we further generalize this non-uniform smoothness condition and develop a simple, yet powerful analysis technique that bounds the gradients along the trajectory, thereby leading to stronger results for both convex and non-convex optimization problems. In particular, we obtain the classical convergence rates for (stochastic) gradient descent and Nesterov's accelerated gradient method in the convex and/or non-convex setting under this general smoothness condition. The new analysis approach does not require gradient clipping and allows heavy-tailed noise with bounded variance in the stochastic setting.

Optimization and Control,Machine Learning

Rescaled Pure Greedy Algorithm for Convex Optimization

A GLOBALLY AND SUPERLIEARLY CONVERGENT FEASIBLE SQP ALGORITHM FOR DEGENERATE CONSTRAINED OPTIMIZATION

Biorthogonal Greedy Algorithms in convex optimization

Convergence and rate of convergence of some greedy algorithms in convex optimization

On the rate of convergence of greedy algorithms

A Randomized Nonlinear Rescaling Method in Large-Scale Constrained Convex Optimization

Greedy Approximation in Convex Optimization

A unified way of analyzing some greedy algorithms

Linear Convergence of Stochastic Iterative Greedy Algorithms With Sparse Constraints

Greedy Algorithms for Cone Constrained Optimization with Convergence Guarantees

Explicit Convergence Rates of Greedy and Random Quasi-Newton Methods

Entropy-based convergence rates of greedy algorithms

A Proximal-Gradient Method for Constrained Optimization

Greedy expansions in convex optimization

A Hybrid Method of Combinatorial Search and Coordinate Descent for Discrete Optimization

Analyzing and Improving Greedy 2-Coordinate Updates for Equality-Constrained Optimization via Steepest Descent in the 1-Norm

Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence.

Greedy Algorithms for Reduced Bases in Banach Spaces

Convex and Non-convex Optimization Under Generalized Smoothness