Abstract:SIAM Journal on Optimization, Volume 31, Issue 3, Page 2141-2170, January 2021. In this work, we consider a constrained convex problem with linear inequalities and provide an inexact penalty reformulation of the problem. The novelty is in the choice of the penalty functions, which are smooth and can induce a nonzero penalty over some points in a feasible region of the original constrained problem. The resulting unconstrained penalized problem is parametrized by two penalty parameters which control the slope and the curvature of the penalty function. With a suitable selection of these penalty parameters, we show that the solutions of the resulting penalized unconstrained problem are feasible for the original constrained problem, under some assumptions. Also, we establish that, with suitable choices of penalty parameters, the solutions of the penalized unconstrained problem can achieve a suboptimal value which is arbitrarily close to the optimal value of the original constrained problem. For the problems with a large number of linear inequality constraints, a particular advantage of such a smooth penalty-based reformulation is that it renders a penalized problem suitable for the implementation of fast incremental gradient methods, which require only one sample from the inequality constraints at each iteration. We consider applying SAGA proposed in [A. Defazio, F. Bach, and S. Lacoste-Julien, SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives, in Proceedings of NIPS, 2014, pp. 1646--1654] to solve the resulting penalized unconstrained problem. Moreover, we propose an alternative approach using a sequence of penalized problem. The approach is based on time-varying penalty parameters and, thus, does not require knowledge of some problem-specific constants that may be difficult to estimate. For a strongly convex objective function, under some conditions on the penalty parameters, we prove that a single-loop full gradient-based algorithm applied to the corresponding time-varying penalized problems converges to the solution of the original constrained problem.

A Penalty-Free Infeasible Approach for a Class of Nonsmooth Optimization Problems Over the Stiefel Manifold

A Novel Penalty Approach for Nonlinear Dynamic Optimization Problems with Inequality Path Constraints

Nonsmooth Optimization over the Stiefel Manifold and Beyond: Proximal Gradient Method and Recent Variants

Error bound and exact penalty method for optimization problems with nonnegative orthogonal constraint

Proximal methods for structured nonsmooth optimization over Riemannian submanifolds

A class of smooth exact penalty function methods for optimization problems with orthogonality constraints

An Exact Penalty Approach for Optimization with Nonnegative Orthogonality Constraints

A nonsmooth exact penalty method for equality-constrained optimization: complexity and implementation

Riemannian Stochastic Proximal Gradient Methods for Nonsmooth Optimization over the Stiefel Manifold

Penalty Methods for a Class of Non-Lipschitz Optimization Problems

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

A Proximal-Gradient Method for Constrained Optimization

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

A Nonconvex Proximal Bundle Method for Nonsmooth Constrained Optimization

A Theory of the NEPv Approach for Optimization On the Stiefel Manifold

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Randomized Submanifold Subgradient Method for Optimization over Stiefel Manifolds

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

A Smooth Inexact Penalty Reformulation of Convex Problems with Linear Constraints

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization