Abstract:The alternating direction method of multipliers (ADM or ADMM) breaks a complex optimization problem into much simpler subproblems. The ADM algorithms are typically short and easy to implement yet exhibit (nearly) state-of-the-art performance for large-scale optimization problems. To apply ADM, we first formulate a given problem into the "ADM-ready" form, so the final algorithm depends on the formulation. A problem like $\mbox{minimize}_\mathbf{x} u(\mathbf{x}) + v(\mathbf{C}\mathbf{x})$ has six different "ADM-ready" formulations. They can be in the primal or dual forms, and they differ by how dummy variables are introduced. To each "ADM-ready" formulation, ADM can be applied in two different orders depending on how the primal variables are updated. Finally, we get twelve different ADM algorithms! How do they compare to each other? Which algorithm should one choose? In this paper, we show that many of the different ways of applying ADM are equivalent. Specifically, we show that ADM applied to a primal formulation is equivalent to ADM applied to its Lagrange dual; ADM is equivalent to a primal-dual algorithm applied to the saddle-point formulation of the same problem. These results are surprising since the primal and dual variables in ADM are seemingly treated very differently, and some previous work exhibit preferences in one over the other on specific problems. In addition, when one of the two objective functions is quadratic, possibly subject to an affine constraint, we show that swapping the update order of the two primal variables in ADM gives the same algorithm. These results identify the few truly different ADM algorithms for a problem, which generally have different forms of subproblems from which it is easy to pick one with the most computationally friendly subproblems.

On the dual step length of the alternating direction method of multipliers

A Golden Ratio Proximal Alternating Direction Method of Multipliers for Separable Convex Optimization

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Accelerated Alternating Direction Method of Multipliers: an Optimal O(1/K) Nonergodic Analysis

Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization

Dual Descent Augmented Lagrangian Method and Alternating Direction Method of Multipliers

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

An Alternating Direction Method of Multipliers with the BFGS Update for Structured Convex Quadratic Optimization

Parallel Direction Method of Multipliers

Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems

Self Equivalence of the Alternating Direction Method of Multipliers

On convergence rates of proximal alternating direction method of multipliers

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints

Understanding the ADMM Algorithm via High-Resolution Differential Equations

The Direct Extension of ADMM for Multi-Block Convex Minimization Problems is Not Necessarily Convergent

The symmetric ADMM with indefinite proximal regularization and its application

General Optimal Step-size for ADMM-type Algorithms: Domain Parametrization and Optimal Rates

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

A Unified Alternating Direction Method of Multipliers by Majorization Minimization