Abstract:SIAM Journal on Optimization, Volume 34, Issue 2, Page 1679-1707, June 2024. Classical primal-dual algorithms attempt to solve [math] by alternately minimizing over the primal variable [math] through primal descent and maximizing the dual variable [math] through dual ascent. However, when [math] is highly nonconvex with complex constraints in [math], the minimization over [math] may not achieve global optimality and, hence, the dual ascent step loses its valid intuition. This observation motivates us to propose a new class of primal-dual algorithms for nonconvex constrained optimization with the key feature to reverse dual ascent to a conceptually new dual descent, in a sense, elevating the dual variable to the same status as the primal variable. Surprisingly, this new dual scheme achieves some best iteration complexities for solving nonconvex optimization problems. In particular, when the dual descent step is scaled by a fractional constant, we name it scaled dual descent (SDD), otherwise, unscaled dual descent (UDD). For nonconvex multiblock optimization with nonlinear equality constraints, we propose SDD-alternating direction method of multipliers (SDD-ADMM) and show that it finds an [math]-stationary solution in [math] iterations. The complexity is further improved to [math] and [math] under proper conditions. We also propose UDD-augmented Lagrangian method (UDD-ALM), combining UDD with ALM, for weakly convex minimization over affine constraints. We show that UDD-ALM finds an [math]-stationary solution in [math] iterations. These complexity bounds for both algorithms either achieve or improve the best-known results in the ADMM and ALM literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing ADMM frameworks.

On the Convergence of Augmented Lagrangian Methods for Constrained Global Optimization

Convergence analysis of primal-dual augmented Lagrangian methods and duality theory

Convergence Properties of Modified and Partially-Augmented Lagrangian Methods for Mathematical Programs with Complementarity Constraints

convergence rates of (dual-primal) balanced augmented Lagrangian methods for linearly constrained convex programming

Local convergence of augmented Lagrangian methods for composite optimization

On Saddle Points of Augmented Lagrangians for Constrained Nonconvex Optimization

Exponential Convergence of Augmented Primal-dual Gradient Algorithms for Partially Strongly Convex Functions

Inexact Augmented Lagrangian Methods for Conic Programs: Quadratic Growth and Linear Convergence

A New Insight on Augmented Lagrangian Method with Applications in Machine Learning

Global Convergence Analysis of the Power Proximal Point and Augmented Lagrangian Method

A New Penalty Dual-Primal Augmented Lagrangian Method and Its Extensions

A Universal Accelerated Primal–Dual Method for Convex Optimization Problems

Convergence analysis of approximate primal solutions in dual first-order methods

A Primal Majorized Semismooth Newton-CG Augmented Lagrangian Method for Large-Scale Linearly Constrained Convex Programming

Learning Constrained Optimization with Deep Augmented Lagrangian Methods

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Augmented Lagrangian method for nonlinear circular conic programs: a local convergence analysis

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

An Augmented Lagrangian Primal-Dual Semismooth Newton Method for Multi-Block Composite Optimization

An Augmented Lagrangian Method for State Constrained Linear Parabolic Optimal Control Problems

The inexact power augmented Lagrangian method for constrained nonconvex optimization