Abstract:This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize $\sum_{i=1}^N f_i(x_i)$ subject to $\sum_{i=1}^N A_i x_i=c, x_i\in \mathcal{X}_i$. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices $A_i$ are mutually near-orthogonal and have full column-rank. Secondly, for general matrices $A_i$, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

Parallel Direction Method of Multipliers

Linearized Block-Wise Alternating Direction Method of Multipliers for Multiple-Block Convex Programming

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Parallel Multi-Block ADMM with O (1 / K ) Convergence

A Hybrid Type ADMM for Multi-Block Separable Convex Programming

An inertial alternating direction method of multipliers for solving a two-block separable convex minimization problem

Two-block vs. Multi-block ADMM: An empirical evaluation of convergence

Alternating Direction Method of Multipliers with Difference of Convex Functions.

An Inertial Proximal Partially Symmetric ADMM-based Algorithm for Linearly Constrained Multi-Block Nonconvex Optimization Problems with Applications.

Convergence of ADMM for Multi-Block Nonconvex Separable Optimization Models

A PROXIMAL ALTERNATING DIRECTION METHOD FOR MULTI-BLOCK COUPLED CONVEX OPTIMIZATION

A Three-Operator Splitting Perspective of a Three-Block ADMM for Convex Quadratic Semidefinite Programming and Beyond

A Generalization of Linearized Alternating Direction Method of Multipliers for Solving Two-Block Separable Convex Programming

A Generalized Alternating Direction Method of Multipliers with Semi-Proximal Terms for Convex Composite Conic Programming

A Linearized Alternating Direction Method of Multipliers with Substitution Procedure.

Modified Alternating Direction Method of Multipliers for Convex Quadratic Semidefinite Programming

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

A linear algebra perspective on the random multi-block ADMM: the QP case

On the O(1/k) Convergence and Parallelization of the Alternating Direction Method of Multipliers

Optimal linearized symmetric ADMM for multi-block separable convex programming