Abstract:We present a new feasible proximal gradient method for constrained optimization where both the objective and constraint functions are given by the summation of a smooth, possibly nonconvex function and a convex simple function. The algorithm converts the original problem into a sequence of convex subproblems. Formulating those subproblems requires the evaluation of at most one gradient value of the original objective and constraint functions. Either exact or approximate subproblem solutions can be computed efficiently in many cases. An important feature of the algorithm is the constraint level parameter. By carefully increasing this level for each subproblem, we provide a simple solution to overcome the challenge of bounding the Lagrangian multipliers and show that the algorithm follows a strictly feasible solution path till convergence to the stationary point. We develop a simple, proximal gradient descent type analysis, showing that the complexity bound of this new algorithm is comparable to gradient descent for the unconstrained setting, which is new in the literature. Exploiting this new design and analysis technique, we extend our algorithms to some more challenging constrained optimization problems where 1) the objective is a stochastic or finite-sum function, and 2) structured nonsmooth functions replace smooth components of both objective and constraint functions. Complexity results for these problems also seem to be new in the literature. Finally, our method can also be applied to convex function-constrained problems where we show complexities similar to the proximal gradient method.

A New Lagrangian-Based First-Order Method for Nonconvex Constrained Optimization

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A Fast Single-Loop Primal-Dual Algorithm for Non-Convex Functional Constrained Optimization

Level Constrained First Order Methods for Function Constrained Optimization

First-Order Methods for Constrained Convex Programming Based on Linearized Augmented Lagrangian Function

First-Order Methods for Nonsmooth Nonconvex Functional Constrained Optimization with or without Slater Points

Developing Lagrangian-based Methods for Nonsmooth Nonconvex Optimization

Global Optimization Algorithm Based on Lagrangian Dual

A First-Order Primal-Dual Method for Nonconvex Constrained Optimization Based on the Augmented Lagrangian

An Augmented Lagrangian Method for Non-Lipschitz Nonconvex Programming

A Modified Lagrangian Algorithm for Solving Nonlinear Constrained Optimization Problems

A Class of Nonlinear Lagrangians: Theory and Algorithm.

Nonconvex Lagrangian-Based Optimization: Monitoring Schemes and Global Convergence

A class of nonlinear Lagrangians for nonconvex second order cone programming

A first-order augmented Lagrangian method for constrained minimax optimization

Accelerated First-Order Optimization under Nonlinear Constraints

First-order algorithms for robust optimization problems via convex-concave saddle-point Lagrangian reformulation

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

The Dual Algorithm Based on a Class of Nonlinear Lagrangians for Nonlinear Programming

Faster Accelerated First-order Methods for Convex Optimization with Strongly Convex Function Constraints

First-Order Methods for Linearly Constrained Bilevel Optimization