A proximal point algorithm for nonsmooth stochastic programming problems

Yigui Ou,Haichan Lin,Zhigang Wang

2008-01-01

Abstract:Based oil the proximal point approach, this paper proposes a practical algorithm for a stochastic programming problem with a nonsmooth objective. Under some conditions the method is proven to be convergent with probability 1 to stationary points. The numerical experience demonstrates the reliability of the method.

What problem does this paper attempt to address?

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

Liwei Wang,Xinggao Liu,Zeyin Zhang

DOI: https://doi.org/10.1080/0305215x.2016.1176828

IF: 2.5

2016-01-01

Engineering Optimization

Abstract:An efficient primal-dual interior-point algorithm using a new non-monotone line esearch filter method is presented for nonlinear constrained programming, which is widely applied in engineering optimization. The new non-monotone line search technique is introduced to lead to relaxed step acceptance conditions and improved convergence performance. It can also avoid the choice of the upper bound on the memory, which brings obvious disadvantages to traditional techniques. Under mild assumptions, the global convergence of the new non-monotone line search filter method is analysed, and fast local convergence is ensured by second order corrections. The proposed algorithm is applied to the classical alkylation process optimization problem and the results illustrate its effectiveness. Some comprehensive comparisons to existing methods are also presented.
A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

Ya-xiang Yuan,Yi Zhang

2024-12-12

Abstract:The proximal point algorithm plays a central role in non-smooth convex programming. The Augmented Lagrangian Method, one of the most famous optimization algorithms, has been found to be closely related to the proximal point algorithm. Due to its importance, accelerated variants of the proximal point algorithm have received considerable attention. In this paper, we first study an Ordinary Differential Equation (ODE) system, which provides valuable insights into proving the convergence rate of the desired algorithm. Using the Lyapunov function technique, we establish the convergence rate of the ODE system. Next, we apply the Symplectic Euler Method to discretize the ODE system to derive a new proximal point algorithm, called the Symplectic Proximal Point Algorithm (SPPA). By utilizing the proof techniques developed for the ODE system, we demonstrate the convergence rate of the SPPA. Additionally, it is shown that existing accelerated proximal point algorithm can be considered a special case of the SPPA in a specific manner. Furthermore, under several additional assumptions, we prove that the SPPA exhibits a finer convergence rate.

Optimization and Control
A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Li, Zhize,Li, Jian

2018-01-01

Abstract:We analyze stochastic gradient algorithms for optimizing nonconvex, nonsmooth finite-sum problems. In particular, the objective function is given by the summation of a differentiable (possibly nonconvex) component, together with a possibly non-differentiable but convex component. We propose a proximal stochastic gradient algorithm based on variance reduction, called ProxSVRG+. Our main contribution lies in the analysis of ProxSVRG+. It recovers several existing convergence results and improves/generalizes them (in terms of the number of stochastic gradient oracle calls and proximal oracle calls). In particular, ProxSVRG+ generalizes the best results given by the SCSG algorithm, recently proposed by [Lei et al., 2017] for the smooth nonconvex case. ProxSVRG+ is also more straightforward than SCSG and yields simpler analysis. Moreover, ProxSVRG+ outperforms the deterministic proximal gradient descent (ProxGD) for a wide range of minibatch sizes, which partially solves an open problem proposed in [Reddi et al., 2016]. Also, ProxSVRG+ uses much less proximal oracle calls than ProxSVRG [Reddi et al., 2016]. Moreover, for nonconvex functions satisfied Polyak-Lojasiewicz condition, we prove that ProxSVRG+ achieves a global linear convergence rate without restart unlike ProxSVRG. Thus, it can automatically switch to the faster linear convergence in some regions as long as the objective function satisfies the PL condition locally in these regions. Finally, we conduct several experiments and the experimental results are consistent with the theoretical results.
N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

Luo Luo,Zhihua Zhang

2018-01-01

Abstract:In this paper, we discuss the problem of minimizing the sum of tw convex functions: a smooth function plus a non-smooth function. Further, the smooth pa rt can be expressed by the average of a large number of smooth component functions, and the non-smooth pa r is equipped with a simple proximal mapping. We propose a proximal stochastic second-order met hod, which is efficient and scalable. It incorporates the Hessian in the smooth part of the function a nd exploits multistage scheme to reduce the variance of the stochastic gradient. We prove that our metho d can achieve linear rate of convergence.
A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

Weijie Liu,Aryan Mokhtari,Asuman Ozdaglar,Sarath Pattathil,Zebang Shen,Nenggan Zheng

2020-01-01

Abstract:In this paper, we focus on solving a class of constrained non-convex non-concave saddle point problems in a decentralized manner by a group of nodes in a network. Specifically, we assume that each node has access to a summand of a global objective function and nodes are allowed to exchange information only with their neighboring nodes. We propose a decentralized variant of the proximal point method for solving this problem. We show that when the objective function is ρ-weakly convex-weakly concave the iterates converge to approximate stationarity with a rate of O(1/ √ T ) where the approximation error depends linearly on √ ρ. We further show that when the objective function satisfies the Minty VI condition (which generalizes the convex-concave case) we obtain convergence to stationarity with a rate of O(1/ √ T ). To the best of our knowledge, our proposed method is the first decentralized algorithm with theoretical guarantees for solving a non-convex non-concave decentralized saddle point problem.
Proximal Point Algorithms for Nonsmooth Convex Optimization with Fixed Point Constraints

Hideaki Iiduka

DOI: https://doi.org/10.48550/arXiv.1602.01932

2016-02-05

Abstract:The problem of minimizing the sum of nonsmooth, convex objective functions defined on a real Hilbert space over the intersection of fixed point sets of nonexpansive mappings, onto which the projections cannot be efficiently computed, is considered. The use of proximal point algorithms that use the proximity operators of the objective functions and incremental optimization techniques is proposed for solving the problem. With the focus on fixed point approximation techniques, two algorithms are devised for solving the problem. One blends an incremental subgradient method, which is a useful algorithm for nonsmooth convex optimization, with a Halpern-type fixed point iteration algorithm. The other is based on an incremental subgradient method and the Krasnosel'ski\uı-Mann fixed point algorithm. It is shown that any weak sequential cluster point of the sequence generated by the Halpern-type algorithm belongs to the solution set of the problem and that there exists a weak sequential cluster point of the sequence generated by the Krasnosel'ski\uı-Mann-type algorithm, which also belongs to the solution set. Numerical comparisons of the two proposed algorithms with existing subgradient methods for concrete nonsmooth convex optimization show that the proposed algorithms achieve faster convergence.

Optimization and Control
A Stochastic Proximal Point Algorithm for Saddle-Point Problems

Luo Luo,Chen Cheng,Li Yujun,Xie Guangzeng,Zhang Zhihua

2019-01-01

Abstract: We consider saddle point problems which objective functions are the average of $n$ strongly convex-concave individual components. Recently, researchers exploit variance reduction methods to solve such problems and achieve linear-convergence guarantees. However, these methods have a slow convergence when the condition number of the problem is very large. In this paper, we propose a stochastic proximal point algorithm, which accelerates the variance reduction method SAGA for saddle point problems. Compared with the catalyst framework, our algorithm reduces a logarithmic term of condition number for the iteration complexity. We adopt our algorithm to policy evaluation and the empirical results show that our method is much more efficient than state-of-the-art methods.
The stochastic proximal distance algorithm

Haoyu Jiang,Jason Xu

DOI: https://doi.org/10.1007/s11222-024-10524-3

IF: 2.3241

2024-11-02

Statistics and Computing

Abstract:Stochastic versions of proximal methods have gained much attention in statistics and machine learning. These algorithms tend to admit simple, scalable forms, and enjoy numerical stability via implicit updates. In this work, we propose and analyze a stochastic version of the recently proposed proximal distance algorithm, a class of iterative optimization methods that recover a desired constrained estimation problem as a penalty parameter . By uncovering connections to related stochastic proximal methods and interpreting the penalty parameter as the learning rate, we justify heuristics used in practical manifestations of the proximal distance method, establishing their convergence guarantees for the first time. Moreover, we extend recent theoretical devices to establish finite error bounds and a complete characterization of convergence rates regimes. We validate our analysis via a thorough empirical study, also showing that unsurprisingly, the proposed method outpaces batch versions on popular learning tasks.

statistics & probability,computer science, theory & methods
A Unified Theory of Stochastic Proximal Point Methods without Smoothness

Peter Richtárik,Abdurakhmon Sadiev,Yury Demidovich

2024-05-25

Abstract:This paper presents a comprehensive analysis of a broad range of variations of the stochastic proximal point method (SPPM). Proximal point methods have attracted considerable interest owing to their numerical stability and robustness against imperfect tuning, a trait not shared by the dominant stochastic gradient descent (SGD) algorithm. A framework of assumptions that we introduce encompasses methods employing techniques such as variance reduction and arbitrary sampling. A cornerstone of our general theoretical approach is a parametric assumption on the iterates, correction and control vectors. We establish a single theorem that ensures linear convergence under this assumption and the $\mu$-strong convexity of the loss function, and without the need to invoke smoothness. This integral theorem reinstates best known complexity and convergence guarantees for several existing methods which demonstrates the robustness of our approach. We expand our study by developing three new variants of SPPM, and through numerical experiments we elucidate various properties inherent to them.

Optimization and Control,Machine Learning
New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

Andrei Patrascu

DOI: https://doi.org/10.48550/arXiv.1901.08663

2020-05-02

Abstract:Large sectors of the recent optimization literature focused in the last decade on the development of optimal stochastic first order schemes for constrained convex models under progressively relaxed assumptions. Stochastic proximal point is an iterative scheme born from the adaptation of proximal point algorithm to noisy stochastic optimization, with a resulting iteration related to stochastic alternating projections. Inspired by the scalability of alternating projection methods, we start from the (linear) regularity assumption, typically used in convex feasiblity problems to guarantee the linear convergence of stochastic alternating projection methods, and analyze a general weak linear regularity condition which facilitates convergence rate boosts in stochastic proximal point schemes. Our applications include many non-strongly convex functions classes often used in machine learning and statistics. Moreover, under weak linear regularity assumption we guarantee $\mathcal{O}\left(\frac{1}{k}\right)$ convergence rate for SPP, in terms of the distance to the optimal set, using only projections onto a simple component set. Linear convergence is obtained for interpolation setting, when the optimal set of the expected cost is included into the optimal sets of each functional component.

Optimization and Control,Machine Learning
A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

Digvijay Boob,Qi Deng,Guanghui Lan,Yilin Wang

DOI: https://doi.org/10.48550/arXiv.2010.12169

2020-10-23

Abstract:Nonconvex sparse models have received significant attention in high-dimensional machine learning. In this paper, we study a new model consisting of a general convex or nonconvex objectives and a variety of continuous nonconvex sparsity-inducing constraints. For this constrained model, we propose a novel proximal point algorithm that solves a sequence of convex subproblems with gradually relaxed constraint levels. Each subproblem, having a proximal point objective and a convex surrogate constraint, can be efficiently solved based on a fast routine for projection onto the surrogate constraint. We establish the asymptotic convergence of the proposed algorithm to the Karush-Kuhn-Tucker (KKT) solutions. We also establish new convergence complexities to achieve an approximate KKT solution when the objective can be smooth/nonsmooth, deterministic/stochastic and convex/nonconvex with complexity that is on a par with gradient descent for unconstrained optimization problems in respective cases. To the best of our knowledge, this is the first study of the first-order methods with complexity guarantee for nonconvex sparse-constrained problems. We perform numerical experiments to demonstrate the effectiveness of our new model and efficiency of the proposed algorithm for large scale problems.

Optimization and Control,Machine Learning
Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Hu Yaohua,Li Chong,Wang Jinhua,Yang Xiaoqi,Zhu Linglingzhi

DOI: https://doi.org/10.1007/s00245-022-09957-x

2023-01-01

Abstract:We propose an inexact linearized proximal algorithm with an adaptive stepsize, together with its globalized version based on the backtracking line-search, to solve a convex composite optimization problem. Under the assumptions of local weak sharp minima of order p (p≥ 1) for the outer convex function and a quasi-regularity condition for the inclusion problem associated to the inner function, we establish superlinear/quadratic convergence results for proposed algorithms. Compared to the linearized proximal algorithms with a constant stepsize proposed in Hu et al. (SIAM J Optim 26(2):1207–1235, 2016), our algorithms own broader applications and higher convergence rates, and the idea of analysis used in the present paper deviates significantly from that of Hu et al. (2016). Numerical applications to the nonnegative inverse eigenvalue problem and the wireless sensor network localization problem indicate that the proposed algorithms are more efficient and robust, and outperform the algorithms in Hu et al. (2016) and some popular algorithms for relevant problems.
A Two-Step Proximal Point Algorithm for Nonconvex Equilibrium Problems with Applications to Fractional Programming

Alfredo Iusem,Felipe Lara,Raúl T. Marcavillaca,Le Hai Yen

DOI: https://doi.org/10.1007/s10898-024-01419-8

2024-07-18

Journal of Global Optimization

Abstract:We present a proximal point type algorithm tailored for tackling pseudomonotone equilibrium problems in a Hilbert space which are not necessarily convex in the second argument of the involved bifunction. Motivated by the extragradient algorithm, we propose a two-step method and we prove that the generated sequence converges strongly to a solution of the nonconvex equilibrium problem under mild assumptions and, also, we establish a linear convergent rate for the iterates. Furthermore, we identify a new class of functions that meet our assumptions, and we provide sufficient conditions for quadratic fractional functions to exhibit strong quasiconvexity. Finally, we perform numerical experiments comparing our algorithm against two alternative methods for classes of nonconvex mixed variational inequalities.

mathematics, applied,operations research & management science
A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Andre Milzarek,Fabian Schaipp,Michael Ulbrich

DOI: https://doi.org/10.1137/22M1488181

2024-03-26

Abstract:We develop an implementable stochastic proximal point (SPP) method for a class of weakly convex, composite optimization problems. The proposed stochastic proximal point algorithm incorporates a variance reduction mechanism and the resulting SPP updates are solved using an inexact semismooth Newton framework. We establish detailed convergence results that take the inexactness of the SPP steps into account and that are in accordance with existing convergence guarantees of (proximal) stochastic variance-reduced gradient methods. Numerical experiments show that the proposed algorithm competes favorably with other state-of-the-art methods and achieves higher robustness with respect to the step size selection.

Optimization and Control,Machine Learning
Proximal Point Algorithms with Inertial Extrapolation for Quasi-convex Pseudo-monotone Equilibrium Problems

Chinedu Izuchukwu,Grace N. Ogwo,Yekini Shehu

DOI: https://doi.org/10.1007/s11067-024-09632-4

2024-06-21

Networks and Spatial Economics

Abstract:In this paper, we study the proximal point algorithm with inertial extrapolation to approximate a solution to the quasi-convex pseudo-monotone equilibrium problem. In the proposed algorithm, the inertial parameter is allowed to take both negative and positive values during implementations. The possibility of the choice of negative values for the inertial parameter sheds more light on the range of values of the inertial parameter for the proximal point algorithm. Under standard assumptions, we prove that the sequence of iterates generated by the proposed algorithm converges to a solution of the equilibrium problem when the bifunction is strongly quasi-convex in its second argument. Sublinear and linear rates of convergence are also given under standard conditions. Numerical results are reported for both cases of negative and positive inertial factor of the proposed algorithm and comparison with related algorithm is discussed.

transportation science & technology,operations research & management science
A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

Mengxiang Zhang,Shengjie Li

DOI: https://doi.org/10.1007/s10915-024-02748-2

2024-12-04

Journal of Scientific Computing

Abstract:In the field of machine learning, many large-scale optimization problems can be decomposed into the sum of two functions: a smooth function and a nonsmooth function with a simple proximal mapping. In light of this, our paper introduces a novel variant of the proximal stochastic quasi-Newton algorithm, grounded in three key components: (i) developing an adaptive sampling method that dynamically increases the sample size during the iteration process, thus preventing rapid growth in sample size and mitigating the noise introduced by the stochastic approximation method; (ii) the integration of stochastic line search to ensure a sufficient decrease in the expected value of the objective function; and (iii) a stable update scheme for the stochastic modified limited-memory Broyden-Fletcher-Goldfarb-Shanno algorithm. For a general objective function, it can be proven that the limit points of the generated sequence almost surely converge to stationary points. Furthermore, the convergence rate and the number of required gradient computations for this process have been analyzed. In the case of a strongly convex objective function, a global linear convergence rate can be achieved, and the number of required gradient computations is thoroughly examined. Finally, numerical experiments demonstrate the robustness of the proposed method across various hyperparameter settings, establishing its competitiveness compared to state-of-the-art methods.

mathematics, applied
GLOBAL OPTIMIZATION FOR NON-CONVEX PROGRAMS VIA CONVEX PROXIMAL POINT METHOD

Yuanyi Zhao,Wenxun Xing

DOI: https://doi.org/10.3934/jimo.2022142

2023-01-01

Journal of Industrial and Management Optimization

Abstract:In this study, a convex proximal point algorithm (CPPA) is considered for solving constrained non-convex problems, and new theoretical results are proposed. It is proved that every cluster point of CPPA is a stationary point, and the initial point of CPPA is key to global optimization. Several sufficient conditions for the initial point selection are provided for CPPA to find the global minimum. Motivated by these results, numerical experiments were conducted on non-convex quadratic programming problems with convex quadratic constraints. The performance of CPPAs was compared, with the initial point randomly selected or obtained through the Lagrangian dual problem. The numerical results demonstrate that the quality of the CPPA with the computed Lagrangian dual initial point is much better than that with the random initial point, in terms of the objective function value.
A globally convergent proximal Newton-type method in nonsmooth convex optimization

Boris S. Mordukhovich,Xiaoming Yuan,Shangzhi Zeng,Jin Zhang

DOI: https://doi.org/10.1007/s10107-022-01797-5

IF: 2.7

2022-03-22

Mathematical Programming

Abstract:The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of variational analysis, we establish implementable results on the global convergence of the proposed algorithm as well as its local convergence with superlinear and quadratic rates. For certain structured problems, the obtained local convergence conditions do not require the local Lipschitz continuity of the corresponding Hessian mappings that is a crucial assumption used in the literature to ensure a superlinear convergence of other algorithms of the proximal Newton type. The conducted numerical experiments of solving the regularized logistic regression model illustrate the possibility of applying the proposed algorithm to deal with practically important problems.

mathematics, applied,computer science, software engineering,operations research & management science
Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.

Rui Zhu,Di Niu,Zongpeng Li

2018-01-01

Abstract:We study stochastic algorithms for solving nonconvex optimization problems with a convex yet possibly nonsmooth regularizer, which find wide applications in many practical machine learning applications. However, compared to asynchronous parallel stochastic gradient descent (AsynSGD), an algorithm targeting smooth optimization, the understanding of the behavior of stochastic algorithms for nonsmooth regularized optimization problems is limited, especially when the objective function is nonconvex. To fill this theoretical gap, in this paper, we propose and analyze asynchronous parallel stochastic proximal gradient (Asyn-ProxSGD) methods for nonconvex problems. We establish an ergodic convergence rate of $O(1/\sqrt{K})$ for the proposed Asyn-ProxSGD, where $K$ is the number of updates made on the model, matching the convergence rate currently known for AsynSGD (for smooth problems). To our knowledge, this is the first work that provides convergence rates of asynchronous parallel ProxSGD algorithms for nonconvex problems. Furthermore, our results are also the first to show the convergence of any stochastic proximal methods without assuming an increasing batch size or the use of additional variance reduction techniques. We implement the proposed algorithms on Parameter Server and demonstrate its convergence behavior and near-linear speedup, as the number of workers increases, on two real-world datasets.

A proximal point algorithm for nonsmooth stochastic programming problems

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

N ov 2 01 6 A Proximal Stochastic Quasi-Newton Algorithm

A Decentralized Proximal Point-type Method for Non-convex Non-concave Saddle Point Problems

Proximal Point Algorithms for Nonsmooth Convex Optimization with Fixed Point Constraints

A Stochastic Proximal Point Algorithm for Saddle-Point Problems

The stochastic proximal distance algorithm

A Unified Theory of Stochastic Proximal Point Methods without Smoothness

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

A Two-Step Proximal Point Algorithm for Nonconvex Equilibrium Problems with Applications to Fractional Programming

A Semismooth Newton Stochastic Proximal Point Algorithm with Variance Reduction

Proximal Point Algorithms with Inertial Extrapolation for Quasi-convex Pseudo-monotone Equilibrium Problems

A Proximal Stochastic Quasi-Newton Algorithm with Dynamical Sampling and Stochastic Line Search

GLOBAL OPTIMIZATION FOR NON-CONVEX PROGRAMS VIA CONVEX PROXIMAL POINT METHOD

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.