Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1679-1707, June 2024. Classical primal-dual algorithms attempt to solve [math] by alternately minimizing over the primal variable [math] through primal descent and maximizing the dual variable [math] through dual ascent. However, when [math] is highly nonconvex with complex constraints in [math], the minimization over [math] may not achieve global optimality and, hence, the dual ascent step loses its valid intuition. This observation motivates us to propose a new class of primal-dual algorithms for nonconvex constrained optimization with the key feature to reverse dual ascent to a conceptually new dual descent, in a sense, elevating the dual variable to the same status as the primal variable. Surprisingly, this new dual scheme achieves some best iteration complexities for solving nonconvex optimization problems. In particular, when the dual descent step is scaled by a fractional constant, we name it scaled dual descent (SDD), otherwise, unscaled dual descent (UDD). For nonconvex multiblock optimization with nonlinear equality constraints, we propose SDD-alternating direction method of multipliers (SDD-ADMM) and show that it finds an [math]-stationary solution in [math] iterations. The complexity is further improved to [math] and [math] under proper conditions. We also propose UDD-augmented Lagrangian method (UDD-ALM), combining UDD with ALM, for weakly convex minimization over affine constraints. We show that UDD-ALM finds an [math]-stationary solution in [math] iterations. These complexity bounds for both algorithms either achieve or improve the best-known results in the ADMM and ALM literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing ADMM frameworks.

A dual algorithm for a class of augmented convex models

A splitting primal-dual proximity algorithm for solving composite optimization problems

Exponential Convergence of Augmented Primal-dual Gradient Algorithms for Partially Strongly Convex Functions

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Primal Dual Alternating Proximal Gradient Algorithms for Nonsmooth Nonconvex Minimax Problems with Coupled Linear Constraints

Primal-dual proximal bundle and conditional gradient methods for convex problems

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

A Modified Primal-Dual Algorithm for Structured Convex Optimization with a Lipschitzian Term

A New Insight on Augmented Lagrangian Method with Applications in Machine Learning

A convex combination based primal-dual algorithm with linesearch for general convex-concave saddle point problems

Some Primal-Dual Theory for Subgradient Methods for Strongly Convex Optimization

A New Penalty Dual-Primal Augmented Lagrangian Method and Its Extensions

Anisotropic Proximal Gradient

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

A new primal-dual algorithm for structured convex optimization involving a lipschitzian term

A Fast Single-Loop Primal-Dual Algorithm for Non-Convex Functional Constrained Optimization

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

On proximal augmented Lagrangian based decomposition methods for dual block-angular convex composite programming problems

Convergence analysis of primal-dual augmented Lagrangian methods and duality theory

A dual gradient-projection method for large-scale strictly convex quadratic problems

A Unified Primal-Dual Algorithm Framework for Inequality Constrained Problems