The proximal map of the weighted mean absolute error

Lukas Baumgärtner,Roland Herzog,Stephan Schmidt,Manuel Weiß

DOI: https://doi.org/10.1002/pamm.202300042

2023-09-23

PAMM

Abstract:Abstract We investigate the proximal map for the weighted mean absolute error function. An algorithm for its efficient and vectorized evaluation is presented. As a demonstration, this algorithm is applied to a nonsmooth energy minimization problem.

English Else

What problem does this paper attempt to address?

An Inexact Riemannian Proximal Gradient Method

Wen Huang,Ke Wei

DOI: https://doi.org/10.48550/arXiv.2111.07269

2021-11-14

Abstract:This paper considers the problem of minimizing the summation of a differentiable function and a nonsmooth function on a Riemannian manifold. In recent years, proximal gradient method and its invariants have been generalized to the Riemannian setting for solving such problems. Different approaches to generalize the proximal mapping to the Riemannian setting lead versions of Riemannian proximal gradient methods. However, their convergence analyses all rely on solving their Riemannian proximal mapping exactly, which is either too expensive or impracticable. In this paper, we study the convergence of an inexact Riemannian proximal gradient method. It is proven that if the proximal mapping is solved sufficiently accurately, then the global convergence and local convergence rate based on the Riemannian Kurdyka-Łojasiewicz property can be guaranteed. Moreover, practical conditions on the accuracy for solving the Riemannian proximal mapping are provided. As a byproduct, the proximal gradient method on the Stiefel manifold proposed in~[CMSZ2020] can be viewed as the inexact Riemannian proximal gradient method provided the proximal mapping is solved to certain accuracy. Finally, numerical experiments on sparse principal component analysis are conducted to test the proposed practical conditions.

Optimization and Control
Anisotropic Proximal Gradient

Emanuel Laude,Panagiotis Patrinos

2024-04-07

Abstract:This paper studies a novel algorithm for nonconvex composite minimization which can be interpreted in terms of dual space nonlinear preconditioning for the classical proximal gradient method. The proposed scheme can be applied to additive composite minimization problems whose smooth part exhibits an anisotropic descent inequality relative to a reference function. It is proved that the anisotropic descent property is closed under pointwise average if the dual Bregman distance is jointly convex and, more specifically, closed under pointwise conic combinations for the KL-divergence. We analyze the method's asymptotic convergence and prove its linear convergence under an anisotropic proximal gradient dominance condition. Applications are discussed including exponentially regularized LPs and logistic regression with nonsmooth regularization. In numerical experiments we show significant improvements of the proposed method over its Euclidean counterparts.

Mathematics
On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Yaohua Hu,Chong Li,Xiaoqi Yang

DOI: https://doi.org/10.1137/140993090

IF: 2.763

2016-01-01

SIAM Journal on Optimization

Abstract:In the present paper, we investigate a linearized proximal algorithm (LPA) for solving a convex composite optimization problem.Each iteration of the LPA is a proximal minimization of the convex composite function with the inner function being linearizedat the current iterate. The LPA has the attractive computational advantage that the solution of each subproblem is a singleton,which avoids the difficulty as in the Gauss--Newton method (GNM) of finding a solution with minimum norm among the set of minimaof its subproblem, while still maintaining the same local convergence rate as that of the GNM. Under the assumptions of localweak sharp minima of order $p$ ($p \in [1,2]$) and a quasi-regularity condition, we establish a local superlinear convergencerate for the LPA. We also propose a globalization strategy for the LPA based on a backtracking line-search and an inexact versionof the LPA. We further apply the LPA to solve a (possibly nonconvex) feasibility problem, as well as a sensor network localizationproblem. Our numerical results illustrate that the LPA meets the demand for an efficient and robust algorithm for the sensor networklocalization problem.
A Riemannian Proximal Newton Method

Wutao Si,P.-A. Absil,Wen Huang,Rujun Jiang,Simon Vary

2024-04-03

Abstract:In recent years, the proximal gradient method and its variants have been generalized to Riemannian manifolds for solving optimization problems with an additively separable structure, i.e., $f + h$, where $f$ is continuously differentiable, and $h$ may be nonsmooth but convex with computationally reasonable proximal mapping. In this paper, we generalize the proximal Newton method to embedded submanifolds for solving the type of problem with $h(x) = \mu \|x\|_1$. The generalization relies on the Weingarten and semismooth analysis. It is shown that the Riemannian proximal Newton method has a local quadratic convergence rate under certain reasonable assumptions. Moreover, a hybrid version is given by concatenating a Riemannian proximal gradient method and the Riemannian proximal Newton method. It is shown that if the switch parameter is chosen appropriately, then the hybrid method converges globally and also has a local quadratic convergence rate. Numerical experiments on random and synthetic data are used to demonstrate the performance of the proposed methods.

Optimization and Control
The Bregman proximal average

Xianfu Wang,Heinz H. Bauschke

DOI: https://doi.org/10.48550/arXiv.2108.11440

2022-02-24

Abstract:We provide a proximal average with repect to a $1$-coercive Legendre function. In the sense of Bregman distance, the Bregman envelope of the proximal average is a convex combination of Bregman envelopes of individual functions. The Bregman proximal mapping of the average is a convex combination of convexified proximal mappings of individual functions. Techniques from variational analysis provide the keys for the Bregman proximal average.

Optimization and Control
Proximal Point Algorithms on Hadamard Manifolds: Linear Convergence and Finite Termination

Jinhua Wang,Chong Li,Genaro Lopez,Jen-Chih Yao

DOI: https://doi.org/10.1137/15m1051257

IF: 2.763

2016-01-01

SIAM Journal on Optimization

Abstract:In the present paper, we consider inexact proximal point algorithms for finding singular points of multivalued vector fields on Hadamard manifolds. The rate of convergence is shown to be linear under the mild assumption of metric subregularity. Furthermore, if the sequence of parameters associated with the iterative scheme converges to 0, then the convergence rate is superlinear. At the same time, the finite termination of the inexact proximal point algorithm is also provided under a weak sharp minima-like condition. Applications to optimization problems are provided. Some of our results are new even in Euclidean spaces, while others improve and/or extend some known results in Euclidean spaces. As a matter of fact, in the case of exact proximal point algorithm, our results improve the corresponding results in [G. C. Bento and J. X. Cruz Neto, Opttm., 63 (2014), pp. 1281-1288]. Finally, several examples are provided to illustrate that our results are applicable while the corresponding results in the Hilbert space setting are not.
On proximal gradient mapping and its minimization in norm via potential function-based acceleration

Beier Chen,Hui Zhang

DOI: https://doi.org/10.48550/arXiv.2212.07149

2022-12-14

Abstract:The proximal gradient descent method, well-known for composite optimization, can be completely described by the concept of proximal gradient mapping. In this paper, we highlight our previous two discoveries of proximal gradient mapping--norm monotonicity and refined descent, with which we are able to extend the recently proposed potential function-based framework from gradient descent to proximal gradient descent.

Optimization and Control
Proximal Algorithms for a class of abstract convex functions

Ewa Bednarczuk,Dirk Lorenz,The Hung Tran

2024-02-27

Abstract:In this paper we analyze a class of nonconvex optimization problem from the viewpoint of abstract convexity. Using the respective generalizations of the subgradient we propose an abstract notion proximal operator and derive a number of algorithms, namely an abstract proximal point method, an abstract forward-backward method and an abstract projected subgradient method. Global convergence results for all algorithms are discussed and numerical examples are given

Optimization and Control
Global convergence of block proximal iteratively reweighted algorithm with extrapolation

Jie Zhang,Xinmin Yang

2023-12-12

Abstract:In this paper, we propose a proximal iteratively reweighted algorithm with extrapolation based on block coordinate update aimed at solving a class of optimization problems which is the sum of a smooth possibly nonconvex loss function and a general nonconvex regularizer with a special structure. The proposed algorithm can be used to solve the $\ell_p(0<p<1)$ regularization problem by employing a updating strategy of the smoothing parameter. It is proved that there exists the nonzero extrapolation parameter such that the objective function is nonincreasing. Moreover, the global convergence and local convergence rate are obtained by using the Kurdyka-Łojasiewicz (KL) property on the objective function. Numerical experiments are given to indicate the efficiency of the proposed algorithm.

Optimization and Control
A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Ting Wang,Hongwei Liu

DOI: https://doi.org/10.1007/s11075-023-01569-y

IF: 2.37

2023-06-30

Numerical Algorithms

Abstract:Extrapolation, restart and stepsize are very powerful strategies for accelerating the convergence rates of first-order algorithms. In this paper, we propose a modified accelerated proximal gradient algorithm (modAPG), which incorporates the adaptive nonmonotone stepsize strategy, extrapolation and a modified rule of setting the extrapolation parameter to zero at some iterations, for solving a minimization problem composed of a smooth (possibly nonconvex) with Lipschitz continuous gradient and nonsmooth (possibly nonconvex) function. Moreover, the proposed algorithm has a simpler form for this problem with convex setting. We show that any cluster point of the sequence generated by modAPG is a critical point of the problem, and analyze the convergence rates of function values and iterates under the assumption that objective function has the Kurdyka-Łojasiewicz property. Finally, we conduct some preliminary numerical experiments for solving the convex problems, such as image deblurring problem and sparse logistic regression problem; and the nonconvex regularization problem, such as the L12\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${L_{\frac{1}{2}}}$$\end{document} penalty and the smoothly clipped absolute deviation (SCAD) penalty problem. Numerical experiments demonstrate the promising performance of the proposed method.

mathematics, applied
Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

Qia Li,Na Zhang,Hanwei Yan

2024-11-24

Abstract:In this paper, we consider a class of structured nonsmooth optimization problems over an embedded submanifold of a Euclidean space, where the first part of the objective is the sum of a difference-of-convex (DC) function and a smooth function, while the remaining part is the square of a weakly convex function over a smooth function. This model problem has many important applications in machine learning and scientific computing, for example, the sparse generalized eigenvalue problem. We propose a manifold proximal-gradient-subgradient algorithm (MPGSA) and show that under mild conditions any accumulation point of the solution sequence generated by it is a critical point of the underlying problem. By assuming the Kurdyka-Łojasiewicz property of an auxiliary function, we further establish the convergence of the full sequence generated by MPGSA under some suitable conditions. When the second component of the DC function involved is the maximum of finite continuously differentiable convex functions, we also propose an enhanced MPGSA with guaranteed subsequential convergence to a lifted B-stationary points of the optimization problem. Finally, some preliminary numerical experiments are conducted to illustrate the efficiency of the proposed algorithms.

Optimization and Control
A Search-Free [math] Homotopy Inexact Proximal-Newton Extragradient Algorithm for Monotone Variational Inequalities

M. Marques Alves,J. M. Pereira,B. F. Svaiter

DOI: https://doi.org/10.1137/23m1593000

IF: 2.763

2024-10-02

SIAM Journal on Optimization

Abstract:SIAM Journal on Optimization, Volume 34, Issue 4, Page 3235-3258, December 2024. We present and study the iteration-complexity of a relative-error inexact proximal-Newton extragradient algorithm for solving smooth monotone variational inequality problems in real Hilbert spaces. We removed a search procedure from Monteiro and Svaiter (2012) by introducing a novel approach based on homotopy, which requires the resolution (at each iteration) of a single strongly monotone linear variational inequality. For a given tolerance [math], our main algorithm exhibits pointwise [math] and ergodic [math] iteration-complexities. From a practical perspective, preliminary numerical experiments indicate that our main algorithm outperforms some previous proximal-Newton schemes.

mathematics, applied
A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Hongwei Liu,Ting Wang,Zexian Liu,Liu, Hongwei,Wang, Ting,Liu, Zexian

DOI: https://doi.org/10.1007/s10898-024-01366-4

2024-03-06

Journal of Global Optimization

Abstract:In this paper, we consider the problem that minimizing the sum of a nonsmooth function with a smooth one in the nonconvex setting, which arising in many contemporary applications such as machine learning, statistics, and signal/image processing. To solve this problem, we propose a new nonmonotone accelerated proximal gradient method with variable stepsize strategy. Note that incorporating inertial term into proximal gradient method is a simple and efficient acceleration technique, while the descent property of the proximal gradient algorithm will lost. In our algorithm, the iterates generated by inertial proximal gradient scheme are accepted when the objective function values decrease or increase appropriately; otherwise, the iteration point is generated by proximal gradient scheme, which makes the function values on a subset of iterates are decreasing. We also introduce a variable stepsize strategy, which does not need a line search or does not need to know the Lipschitz constant and makes the algorithm easy to implement. We show that the sequence of iterates generated by the algorithm converges to a critical point of the objective function. Further, under the assumption that the objective function satisfies the Kurdyka–Łojasiewicz inequality, we prove the convergence rates of the objective function values and the iterates. Moreover, numerical results on both convex and nonconvex problems are reported to demonstrate the effectiveness and superiority of the proposed method and stepsize strategy.

mathematics, applied,operations research & management science
An approximation proximal gradient algorithm for nonconvex-linear minimax problems with nonconvex nonsmooth terms

Jiefei He,Huiling Zhang,Zi Xu

DOI: https://doi.org/10.1007/s10898-024-01383-3

2024-03-26

Journal of Global Optimization

Abstract:Nonconvex minimax problems have attracted significant attention in machine learning, wireless communication and many other fields. In this paper, we propose an efficient approximation proximal gradient algorithm for solving a class of nonsmooth nonconvex-linear minimax problems with a nonconvex nonsmooth term, and the number of iteration to find an -stationary point is upper bounded by . Some numerical results on one-bit precoding problem in massive MIMO system and a distributed non-convex optimization problem demonstrate the effectiveness of the proposed algorithm.

mathematics, applied,operations research & management science
A Decentralized Proximal Gradient Tracking Algorithm for Composite Optimization on Riemannian Manifolds

Lei Wang,Le Bao,Xin Liu

2024-01-22

Abstract:This paper focuses on minimizing a smooth function combined with a nonsmooth regularization term on a compact Riemannian submanifold embedded in the Euclidean space under a decentralized setting. Typically, there are two types of approaches at present for tackling such composite optimization problems. The first, subgradient-based approaches, rely on subgradient information of the objective function to update variables, achieving an iteration complexity of $\mathcal{O}(\epsilon^{-4}\log^2(\epsilon^{-2}))$. The second, smoothing approaches, involve constructing a smooth approximation of the nonsmooth regularization term, resulting in an iteration complexity of $\mathcal{O}(\epsilon^{-4})$. This paper proposes a proximal gradient type algorithm that fully exploits the composite structure. The global convergence to a stationary point is established with a significantly improved iteration complexity of $\mathcal{O}(\epsilon^{-2})$. To validate the effectiveness and efficiency of our proposed method, we present numerical results in real-world applications, showcasing its superior performance.

Optimization and Control
Efficient algorithms for implementing incremental proximal-point methods

Alex Shtoff

2024-06-18

Abstract:Model training algorithms which observe a small portion of the training set in each computational step are ubiquitous in practical machine learning, and include both stochastic and online optimization methods. In the vast majority of cases, such algorithms typically observe the training samples via the gradients of the cost functions the samples incur. Thus, these methods exploit are the slope of the cost functions via their first-order approximations. To address limitations of gradient-based methods, such as sensitivity to step-size choice in the stochastic setting, or inability to use small function variability in the online setting, several streams of research attempt to exploit more information about the cost functions than just their gradients via the well-known proximal operators. However, implementing such methods in practice poses a challenge, since each iteration step boils down to computing the proximal operator, which may not be easy. In this work we devise a novel algorithmic framework, which exploits convex duality theory to achieve both algorithmic efficiency and software modularity of proximal operator implementations, in order to make experimentation with incremental proximal optimization algorithms accessible to a larger audience of researchers and practitioners, by reducing the gap between their theoretical description in research papers and their use in practice. We provide a reference Python implementation for the framework developed in this paper as an open source library at on <a class="link-external link-https" href="https://github.com/alexshtf/inc_prox_pt/releases/tag/prox_pt_paper" rel="external noopener nofollow">this https URL</a>, along with examples which demonstrate our implementation on a variety of problems, and reproduce the numerical experiments in this paper. The pure Python reference implementation is not necessarily the most efficient, but is a basis for creating efficient implementations by combining Python with a native backend.

Machine Learning,Optimization and Control
Local convergence of an inexact proximal algorithm for weakly convex functions

Valdines Leite de Sousa Junior,Lucas Vidal de Meireles,Samara Costa Lima,Gilson do Nascimento Silva

2023-08-04

Abstract:Since introduced by Martinet and Rockafellar, the proximal point algorithm was generalized in many fruitful directions. More recently, in 2002, Pennanen studied the proximal point algorithm without monotonicity. A year later, Iusem and Svaiter joined Pennanen to present inexact variants of the method, again without monotonicity. Building on the foundation laid by these two prior works, we propose a variant of the proximal point algorithm designed specifically for weakly convex functions. Our motivation for introducing this inexact algorithm is to increase its versatility and applicability in a broader range of scenarios in optimization and introduce a more adaptable version of the method for typical generalizations. Our study relies heavily on the Moreau envelope, a well-known mathematical tool used to analyze the behavior of the proximal operator. By leveraging the properties of the Moreau envelope, we are able to prove that the proximal algorithm converges in local contexts. Moreover, we present a complexity result to determine the practical feasibility of the proximal algorithm.

Optimization and Control
A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

Ya-xiang Yuan,Yi Zhang

2024-12-12

Abstract:The proximal point algorithm plays a central role in non-smooth convex programming. The Augmented Lagrangian Method, one of the most famous optimization algorithms, has been found to be closely related to the proximal point algorithm. Due to its importance, accelerated variants of the proximal point algorithm have received considerable attention. In this paper, we first study an Ordinary Differential Equation (ODE) system, which provides valuable insights into proving the convergence rate of the desired algorithm. Using the Lyapunov function technique, we establish the convergence rate of the ODE system. Next, we apply the Symplectic Euler Method to discretize the ODE system to derive a new proximal point algorithm, called the Symplectic Proximal Point Algorithm (SPPA). By utilizing the proof techniques developed for the ODE system, we demonstrate the convergence rate of the SPPA. Additionally, it is shown that existing accelerated proximal point algorithm can be considered a special case of the SPPA in a specific manner. Furthermore, under several additional assumptions, we prove that the SPPA exhibits a finer convergence rate.

Optimization and Control
Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation

Zhao Song,Mingquan Ye,Junze Yin,Lichen Zhang

2023-07-28

Abstract:Weighted low rank approximation is a fundamental problem in numerical linear algebra, and it has many applications in machine learning. Given a matrix $M \in \mathbb{R}^{n \times n}$, a weight matrix $W \in \mathbb{R}_{\geq 0}^{n \times n}$, a parameter $k$, the goal is to output two matrices $U, V \in \mathbb{R}^{n \times k}$ such that $\| W \circ (M - U V^\top) \|_F$ is minimized, where $\circ$ denotes the Hadamard product. Such a problem is known to be NP-hard and even hard to approximate assuming Exponential Time Hypothesis [GG11, RSW16]. Meanwhile, alternating minimization is a good heuristic solution for approximating weighted low rank approximation. The work [LLR16] shows that, under mild assumptions, alternating minimization does provide provable guarantees. In this work, we develop an efficient and robust framework for alternating minimization. For weighted low rank approximation, this improves the runtime of [LLR16] from $n^2 k^2$ to $n^2k$. At the heart of our work framework is a high-accuracy multiple response regression solver together with a robust analysis of alternating minimization.

Machine Learning,Optimization and Control

The proximal map of the weighted mean absolute error

An Inexact Riemannian Proximal Gradient Method

Anisotropic Proximal Gradient

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A Riemannian Proximal Newton Method

The Bregman proximal average

Proximal Point Algorithms on Hadamard Manifolds: Linear Convergence and Finite Termination

On proximal gradient mapping and its minimization in norm via potential function-based acceleration

Proximal Algorithms for a class of abstract convex functions

Global convergence of block proximal iteratively reweighted algorithm with extrapolation

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

A Search-Free [math] Homotopy Inexact Proximal-Newton Extragradient Algorithm for Monotone Variational Inequalities

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

An approximation proximal gradient algorithm for nonconvex-linear minimax problems with nonconvex nonsmooth terms

A Decentralized Proximal Gradient Tracking Algorithm for Composite Optimization on Riemannian Manifolds

Efficient algorithms for implementing incremental proximal-point methods

Local convergence of an inexact proximal algorithm for weakly convex functions

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation