Complexity and numerical experiments of a new adaptive generic proximal bundle method

Vincent Guigues,Renato Monteiro,Benoit Tran

2024-10-20

Abstract:This paper develops an adaptive generic proximal bundle method, shows its complexity, and presents numerical experiments comparing this method with two bundle methods on a set of optimization problems.

Optimization and Control

What problem does this paper attempt to address?

Parameter-free proximal bundle methods with adaptive stepsizes for hybrid convex composite optimization problems

Renato D.C. Monteiro,Honghao Zhang

2024-10-28

Abstract:This paper develops a parameter-free adaptive proximal bundle method with two important features: 1) adaptive choice of variable prox stepsizes that "closely fits" the instance under consideration; and 2) adaptive criterion for making the occurrence of serious steps easier. Computational experiments show that our method performs substantially fewer consecutive null steps (i.e., a shorter cycle) while maintaining the number of serious steps under control. As a result, our method performs significantly less number of iterations than its counterparts based on a constant prox stepsize choice and a non-adaptive cycle termination criterion. Moreover, our method is very robust relative to the user-provided initial stepsize.

Optimization and Control
A single cut proximal bundle method for stochastic convex composite optimization

Jiaming Liang,Vincent Guigues,Renato D.C. Monteiro

2023-10-21

Abstract:This paper considers optimization problems where the objective is the sum of a function given by an expectation and a closed convex composite function, and proposes stochastic composite proximal bundle (SCPB) methods for solving it. Complexity guarantees are established for them without requiring knowledge of parameters associated with the problem instance. Moreover, it is shown that they have optimal complexity when these problem parameters are known. To the best of our knowledge, this is the first proximal bundle method for stochastic programming able to deal with continuous distributions. Finally, we present computational results showing that SCPB substantially outperforms the robust stochastic approximation (RSA) method in all instances considered.

Optimization and Control
A proximal bundle variant with optimal iteration-complexity for a large range of prox stepsizes

Jiaming Liang,Renato D.C. Monteiro

DOI: https://doi.org/10.48550/arXiv.2003.11457

2021-06-08

Abstract:This paper presents a proximal bundle variant, namely, the relaxed proximal bundle (RPB) method, for solving convex nonsmooth composite optimization problems. Like other proximal bundle variants, RPB solves a sequence of prox bundle subproblems whose objective functions are regularized composite cutting-plane models. Moreover, RPB uses a novel condition to decide whether to perform a serious or null iteration which does not necessarily yield a function value decrease. Optimal iteration-complexity bounds for RPB are established for a large range of prox stepsizes, both in the convex and strongly convex settings. To the best of our knowledge, this is the first time that a proximal bundle variant is shown to be optimal for a large range of prox stepsizes. Finally, iteration-complexity results for RPB to obtain iterates satisfying practical termination criteria, rather than near optimal solutions, are also derived.

Optimization and Control
Proximal bundle methods for hybrid weakly convex composite optimization problems

Jiaming Liang,Renato D.C. Monteiro,Honghao Zhang

DOI: https://doi.org/10.48550/arXiv.2303.14896

2023-03-27

Abstract:This paper establishes the iteration-complexity of proximal bundle methods for solving hybrid (i.e., a blend of smooth and nonsmooth) weakly convex composite optimization (HWC-CO) problems. This is done in a unified manner by considering a proximal bundle framework (PBF) based on a generic bundle update scheme which includes various well-known bundle update schemes. In contrast to other wellknown stationary conditions in the context of HWC-CO, PBF uses a new stationarity measure which is easily verifiable and, at the same time, implies any of the former ones.

Optimization and Control
Universal subgradient and proximal bundle methods for convex and strongly convex hybrid composite optimization

Vincent Guigues,Jiaming Liang,Renato D.C. Monteiro

2024-08-03

Abstract:This paper develops two parameter-free methods for solving convex and strongly convex hybrid composite optimization problems, namely, a composite subgradient type method and a proximal bundle type method. Both functional and stationary complexity bounds for the two methods are established in terms of the unknown strong convexity parameter. To the best of our knowledge, the two proposed methods are the first universal methods for solving hybrid strongly convex composite optimization problems that do not rely on any restart scheme nor require the knowledge of the optimal value.

Optimization and Control
Primal-dual proximal bundle and conditional gradient methods for convex problems

Jiaming Liang

2024-12-01

Abstract:This paper studies the primal-dual convergence and iteration-complexity of proximal bundle methods for solving nonsmooth problems with convex structures. More specifically, we develop a family of primal-dual proximal bundle methods for solving convex nonsmooth composite optimization problems and establish the iteration-complexity in terms of a primal-dual gap. We also propose a class of proximal bundle methods for solving convex-concave composite saddle-point problems and establish the iteration-complexity to find an approximate saddle-point. This paper places special emphasis on the primal-dual perspective of the proximal bundle method. In particular, we discover an interesting duality between the condition gradient method and the cutting-plane scheme used within the proximal bundle method.

Optimization and Control
A Proximal-Gradient Method for Constrained Optimization

Yutong Dai,Xiaoyi Qu,Daniel P. Robinson

2024-04-11

Abstract:We present a new algorithm for solving optimization problems with objective functions that are the sum of a smooth function and a (potentially) nonsmooth regularization function, and nonlinear equality constraints. The algorithm may be viewed as an extension of the well-known proximal-gradient method that is applicable when constraints are not present. To account for nonlinear equality constraints, we combine a decomposition procedure for computing trial steps with an exact merit function for determining trial step acceptance. Under common assumptions, we show that both the proximal parameter and merit function parameter eventually remain fixed, and then prove a worst-case complexity result for the maximum number of iterations before an iterate satisfying approximate first-order optimality conditions for a given tolerance is computed. Our preliminary numerical results indicate that our approach has great promise, especially in terms of returning approximate solutions that are structured (e.g., sparse solutions when a one-norm regularizer is used).

Optimization and Control
Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Hu Yaohua,Li Chong,Wang Jinhua,Yang Xiaoqi,Zhu Linglingzhi

DOI: https://doi.org/10.1007/s00245-022-09957-x

2023-01-01

Abstract:We propose an inexact linearized proximal algorithm with an adaptive stepsize, together with its globalized version based on the backtracking line-search, to solve a convex composite optimization problem. Under the assumptions of local weak sharp minima of order p (p≥ 1) for the outer convex function and a quasi-regularity condition for the inclusion problem associated to the inner function, we establish superlinear/quadratic convergence results for proposed algorithms. Compared to the linearized proximal algorithms with a constant stepsize proposed in Hu et al. (SIAM J Optim 26(2):1207–1235, 2016), our algorithms own broader applications and higher convergence rates, and the idea of analysis used in the present paper deviates significantly from that of Hu et al. (2016). Numerical applications to the nonnegative inverse eigenvalue problem and the wireless sensor network localization problem indicate that the proposed algorithms are more efficient and robust, and outperform the algorithms in Hu et al. (2016) and some popular algorithms for relevant problems.
A Nonconvex Proximal Bundle Method for Nonsmooth Constrained Optimization

Jie Shen,Fang-Fang Guo,Na Xu

DOI: https://doi.org/10.1155/2024/5720769

IF: 2.3

2024-02-08

Complexity

Abstract:An implementable algorithm for solving nonsmooth nonconvex constrained optimization is proposed by combining bundle ideas, proximity control, and the exact penalty function. We construct two kinds of approximations to nonconvex objective function; these two approximations correspond to the convex and concave behaviors of the objective function at the current point, which captures precisely the characteristic of the objective function. The penalty coefficients are increased only a finite number of times under the conditions of Slater constraint qualification and the boundedness of the constrained set, which limit the unnecessary penalty growth. The given algorithm converges to an approximate stationary point of the exact penalty function for constrained nonconvex optimization with weakly semismooth objective function. We also provide the results of some preliminary numerical testing to show the validity and efficiency of the proposed method.

mathematics, interdisciplinary applications,multidisciplinary sciences
The Proximal Bundle Algorithm Under a Frank-Wolfe Perspective: an Improved Complexity Analysis

David Fersztand,Xu Andy Sun

2024-11-25

Abstract:The proximal bundle algorithm (PBA) is a fundamental and computationally effective algorithm for solving optimization problems with nonsmooth components. We investigate its convergence rate, focusing on composite settings where one function is smooth and the other is piecewise linear. We interpret a sequence of null steps of the PBA as a Frank-Wolfe algorithm on the Moreau envelope of the dual problem. In light of this correspondence, we first extend the linear convergence of Kelley's method on convex piecewise linear functions from the positive homogeneous to the general case. Building on this result, we propose a novel complexity analysis of PBA and derive a $\mathcal{O}(\epsilon^{-4/5})$ iteration complexity, improving upon the best known $\mathcal{O}(\epsilon^{-2})$ guarantee. This approach also unveils new insights on bundle management.

Optimization and Control
Linear Convergence of the Derivative-Free Proximal Bundle Method on Convex Nonsmooth Functions, with Application to the Derivative-Free -Algorithm

C. Planiden,T. Rajapaksha

DOI: https://doi.org/10.1007/s11228-024-00718-2

2024-05-21

Set-Valued and Variational Analysis

Abstract:Proximal bundle methods are a class of optimisation algorithms that leverage the proximal operator to address nonsmoothness in the objective function efficiently. This study focuses on a derivative-free (DFO) proximal bundle method and one of its applications called the DFO -algorithm. These algorithms incorporate approximate proximal points as subprocedures in order to optimise convex nonsmooth functions based on approximated subdifferential information. Interestingly, the classical -algorithm, which operates on true subgradient values, achieves superlinear convergence. At each iteration, the algorithm divides the whole space into two: the smooth -space and the nonsmooth -space. It takes a Newton-like step on the -space and a proximal-point step on the -space, enabling it to handle both smooth and nonsmooth parts effectively and converge faster. In this work, we reveal the worst possible convergence rate for the DFO -method by showing the linear convergence of the DFO proximal bundle method. This will be done by presenting a suitable framework and using the subdifferential-based error bound on the distance to critical points.

mathematics, applied
An adaptive proximal point algorithm framework and application to large-scale optimization

Meng Lu,Zheng Qu

DOI: https://doi.org/10.48550/arXiv.2008.08784

2021-02-25

Abstract:We investigate the proximal point algorithm (PPA) and its inexact extensions under an error bound condition, which guarantees a global linear convergence if the proximal regularization parameter is larger than the error bound condition parameter. We propose an adaptive generalized proximal point algorithm (AGPPA), which adaptively updates the proximal regularization parameters based on some implementable criteria. We show that AGPPA achieves linear convergence without any knowledge of the error bound condition parameter, and the rate only differs from the optimal one by a logarithm term. We apply AGPPA on convex minimization problem and analyze the iteration complexity bound of the resulting algorithm. Our framework and the complexity results apply to arbitrary linearly convergent inner solver and allows a hybrid with any locally fast convergent method. We illustrate the performance of AGPPA by applying it to solve large-scale linear programming (LP) problem. The resulting complexity bound has a weaker dependence on the Hoffman constant and scales with the dimension better than linearized ADMM. In numerical experiments, our algorithm demonstrates improved performance in obtaining solution of medium accuracy on large-scale LP problem.

Optimization and Control
Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization

Guanghui Lan

DOI: https://doi.org/10.1007/s10107-013-0737-x

IF: 2.7

2013-12-29

Mathematical Programming

Abstract:The main goal of this paper is to develop uniformly optimal first-order methods for convex programming (CP). By uniform optimality we mean that the first-order methods themselves do not require the input of any problem parameters, but can still achieve the best possible iteration complexity bounds. By incorporating a multi-step acceleration scheme into the well-known bundle-level method, we develop an accelerated bundle-level method, and show that it can achieve the optimal complexity for solving a general class of black-box CP problems without requiring the input of any smoothness information, such as, whether the problem is smooth, nonsmooth or weakly smooth, as well as the specific values of Lipschitz constant and smoothness level. We then develop a more practical, restricted memory version of this method, namely the accelerated prox-level (APL) method. We investigate the generalization of the APL method for solving certain composite CP problems and an important class of saddle-point problems recently studied by Nesterov (Math Program 103:127–152, 2005). We present promising numerical results for these new bundle-level methods applied to solve certain classes of semidefinite programming and stochastic programming problems.

mathematics, applied,computer science, software engineering,operations research & management science
Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

Kuangyu Ding,Jingyang Li,Kim-Chuan Toh

2024-10-26

Abstract:Stochastic gradient methods for minimizing nonconvex composite objective functions typically rely on the Lipschitz smoothness of the differentiable part, but this assumption fails in many important problem classes, leading to instability of the algorithms in both theory and practice. To address this, we propose a family of stochastic Bregman proximal gradient (SBPG) methods that only require smooth adaptivity. SBPG replaces the quadratic approximation in SGD with a Bregman proximity measure, offering a better approximation model that handles non-Lipschitz gradients in nonconvex objectives. We establish the convergence properties of vanilla SBPG and show it achieves optimal sample complexity in the nonconvex setting. Experimental results on quadratic inverse problems demonstrate SBPG's robustness in terms of stepsize selection and sensitivity to the initial point. Furthermore, we introduce a momentum-based variant, MSBPG, which enhances convergence by relaxing the mini-batch size requirement while preserving the optimal oracle complexity. We apply a polynomial kernel function based MBPG to the loss function with polynomial growth. Experimental results on benchmark datasets confirm the effectiveness and robustness of MSBPG. Given its negligible additional computational cost compared to SGD in large-scale optimization, MSBPG shows promise as a universal optimizer for future applications.

Optimization and Control,Machine Learning
A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

Qingsong Wang,Zehui Liu,Chunfeng Cui,Deren Han

2024-01-04

Abstract:In this paper, we explore a specific optimization problem that involves the combination of a differentiable nonconvex function and a nondifferentiable function. The differentiable component lacks a global Lipschitz continuous gradient, posing challenges for optimization. To address this issue and accelerate the convergence, we propose a Bregman proximal stochastic gradient method with extrapolation (BPSGE), which only requires smooth adaptivity of the differentiable part. Under the variance reduction framework, we not only analyze the subsequential and global convergence of the proposed algorithm under certain conditions, but also analyze the sublinear convergence rate of the subsequence, and the complexity of the algorithm, revealing that the BPSGE algorithm requires at most O(epsilon\^\,(-2)) iterations in expectation to attain an epsilon-stationary point. To validate the effectiveness of our proposed algorithm, we conduct numerical experiments on three real-world applications: graph regularized nonnegative matrix factorization (NMF), matrix factorization with weakly-convex regularization, and NMF with nonconvex sparsity constraints. These experiments demonstrate that BPSGE is faster than the baselines without extrapolation.

Optimization and Control
Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization

Yan Gu,Nobuo Yamashita

DOI: https://doi.org/10.48550/arXiv.1906.12112

2019-06-28

Abstract:This paper studies a proximal alternating direction method of multipliers (ADMM) with variable metric indefinite proximal terms for linearly constrained convex optimization problems. The proximal ADMM plays an important role in many application areas, since the subproblems of the method are easy to solve. Recently, it is reported that the proximal ADMM with a certain fixed indefinite proximal term is faster than that with a positive semidefinite term, and still has the global convergence property. On the other hand, Gu and Yamashita studied a variable metric semidefinite proximal ADMM whose proximal term is generated by the BFGS update. They reported that a slightly indefinite matrix also makes the algorithm work well in their numerical experiments. Motivated by this fact, we consider a variable metric indefinite proximal ADMM, and give sufficient conditions on the proximal terms for the global convergence. Moreover, we propose a new indefinite proximal term based on the BFGS update which can satisfy the conditions for the global convergence.

Optimization and Control
A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Yongcun Song,Zimeng Wang,Xiaoming Yuan,Hangrui Yue

2024-09-25

Abstract:We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

Optimization and Control
Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

Jian Chen,Liping Tang,Xinmin Yang

2023-08-09

Abstract:When minimizing a multiobjective optimization problem (MOP) using multiobjective gradient descent methods, the imbalances among objective functions often decelerate the convergence. In response to this challenge, we propose two types of the Barzilai-Borwein proximal gradient method for multi-objective composite optimization problems (BBPGMO). We establish convergence rates for BBPGMO, demonstrating that it achieves rates of $O(\frac{1}{\sqrt{k}})$, $O(\frac{1}{k})$, and $O(r^{k})(0<r<1)$ for non-convex, convex, and strongly convex problems, respectively. Furthermore, we show that BBPGMO exhibits linear convergence for MOPs with several linear objective functions. Interestingly, the linear convergence rate of BBPGMO surpasses the existing convergence rates of first-order methods for MOPs, which indicates its enhanced performance and its ability to effectively address imbalances from theoretical perspective. Finally, we provide numerical examples to illustrate the efficiency of the proposed method and verify the theoretical results.

Optimization and Control
Proximal Diagonal Newton Methods for Composite Optimization Problems

Shotaro Yagishita,Shummin Nakayama

2023-10-11

Abstract:This paper proposes new proximal Newton-type methods with a diagonal metric for solving composite optimization problems whose objective function is the sum of a twice continuously differentiable function and a proper closed directionally differentiable function. Although proximal Newton-type methods using diagonal metrics have been shown to be superior to the proximal gradient method numerically, no theoretical results have been obtained to suggest this superiority. Even though our proposed method is based on a simple idea, its convergence rate suggests an advantage over the proximal gradient method in certain situations. Numerical experiments show that our proposed algorithms are effective, especially in the nonconvex case.

Optimization and Control
An Alternating Direction Method of Multipliers with the BFGS Update for Structured Convex Quadratic Optimization

Yan Gu,Nobuo Yamashita

DOI: https://doi.org/10.48550/arXiv.1903.02270

2019-03-06

Abstract:The alternating direction method of multipliers (ADMM) is an effective method for solving wide fields of convex problems. At each iteration, the classical ADMM solves two subproblems exactly. However, in many applications, it is expensive or impossible to obtain the exact solutions of the subproblems. To overcome the difficulty, some proximal terms are added to the subproblems. This class of methods normally solves the original subproblem approximately, and thus takes more iterations. This fact urges us to consider that a special proximal term can lead to a better result as the classical ADMM. In this paper, we propose a proximal ADMM whose regularized matrix in the proximal term is generated by the BFGS update (or limited memory BFGS) at every iteration. These types of matrices use second-order information of the objective function. The convergence of the proposed method is proved under certain assumptions. Numerical results are presented to show the effectiveness of the proposed proximal ADMM.

Optimization and Control

Complexity and numerical experiments of a new adaptive generic proximal bundle method

Parameter-free proximal bundle methods with adaptive stepsizes for hybrid convex composite optimization problems

A single cut proximal bundle method for stochastic convex composite optimization

A proximal bundle variant with optimal iteration-complexity for a large range of prox stepsizes

Proximal bundle methods for hybrid weakly convex composite optimization problems

Universal subgradient and proximal bundle methods for convex and strongly convex hybrid composite optimization

Primal-dual proximal bundle and conditional gradient methods for convex problems

A Proximal-Gradient Method for Constrained Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Nonconvex Proximal Bundle Method for Nonsmooth Constrained Optimization

The Proximal Bundle Algorithm Under a Frank-Wolfe Perspective: an Improved Complexity Analysis

Linear Convergence of the Derivative-Free Proximal Bundle Method on Convex Nonsmooth Functions, with Application to the Derivative-Free -Algorithm

An adaptive proximal point algorithm framework and application to large-scale optimization

Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization

Nonconvex Stochastic Bregman Proximal Gradient Method for Nonconvex Composite Problems

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

Barzilai-Borwein Proximal Gradient Methods for Multiobjective Composite Optimization Problems with Improved Linear Convergence

Proximal Diagonal Newton Methods for Composite Optimization Problems

An Alternating Direction Method of Multipliers with the BFGS Update for Structured Convex Quadratic Optimization