Abstract:Classical primal-dual algorithms attempt to solve $\max_{\mu}\min_{x} \mathcal{L}(x,\mu)$ by alternatively minimizing over the primal variable $x$ through primal descent and maximizing the dual variable $\mu$ through dual ascent. However, when $\mathcal{L}(x,\mu)$ is highly nonconvex with complex constraints in $x$, the minimization over $x$ may not achieve global optimality, and hence the dual ascent step loses its valid intuition. This observation motivates us to propose a new class of primal-dual algorithms for nonconvex constrained optimization with the key feature to reverse dual ascent to a conceptually new dual descent, in a sense, elevating the dual variable to the same status as the primal variable. Surprisingly, this new dual scheme achieves some best iteration complexities for solving nonconvex optimization problems. In particular, when the dual descent step is scaled by a fractional constant, we name it scaled dual descent (SDD), otherwise, unscaled dual descent (UDD). For nonconvex multiblock optimization with nonlinear equality constraints, we propose SDD-ADMM and show that it finds an $\epsilon$-stationary solution in $\mathcal{O}(\epsilon^{-4})$ iterations. The complexity is further improved to $\mathcal{O}(\epsilon^{-3})$ and $\mathcal{O}(\epsilon^{-2})$ under proper conditions. We also propose UDD-ALM, combining UDD with ALM, for weakly convex minimization over affine constraints. We show that UDD-ALM finds an $\epsilon$-stationary solution in $\mathcal{O}(\epsilon^{-2})$ iterations. These complexity bounds for both algorithms either achieve or improve the best-known results in the ADMM and ALM literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing ADMM frameworks.

ADMM for 0/1 D-Opt and MESP relaxations

Convex relaxation for the generalized maximum-entropy sampling problem

An Empirical Study of ADMM for Nonconvex Problems

Optimizing ADMM and Over-Relaxed ADMM Parameters for Linear Quadratic Problems

An outer-approximation algorithm for maximum-entropy sampling

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

A Riemannian ADMM

A partial Bregman ADMM with a general relaxation factor for structured nonconvex and nonsmooth optimization

Relaxing Alternating Direction Method of Multipliers (ADMM) for Linear Inverse Problems

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

ADMM for Nonconvex Optimization under Minimal Continuity Assumption

Moreau Envelope ADMM for Decentralized Weakly Convex Optimization

Nonconvex Zeroth-Order Stochastic ADMM Methods with Lower Function Query Complexity

Dual Descent ALM and ADMM

NysADMM: faster composite convex optimization via low-rank approximation

Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems

A Multi-Level ADMM Algorithm for Elliptic PDE-constrained Optimization Problems

Online Alternating Direction Method (longer version)

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization