Abstract:This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize $\sum_{i=1}^N f_i(x_i)$ subject to $\sum_{i=1}^N A_i x_i=c, x_i\in \mathcal{X}_i$. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices $A_i$ are mutually near-orthogonal and have full column-rank. Secondly, for general matrices $A_i$, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

Convergence Analysis of Parallel Multi-block ADMM Via Discrete-time Recurrent Neural Networks.

Convergence Analysis and Design of Multi-block ADMM via Switched Control Theory.

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

Linearized Block-Wise Alternating Direction Method of Multipliers for Multiple-Block Convex Programming

A Convergent ADMM Framework for Efficient Neural Network Training

Convergence Analysis of L-ADMM for Multi-block Linear-Constrained Separable Convex Minimization Problem

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

Convergence Analysis of Alternating Direction Method of Multipliers for a Class of Separable Convex Programming

Parallel Multi-Block ADMM with O (1 / K ) Convergence

Parallel Direction Method of Multipliers

Convergence of Multi-Block Bregman ADMM for Nonconvex Composite Problems

On the O(1/k) Convergence and Parallelization of the Alternating Direction Method of Multipliers

Two-block vs. Multi-block ADMM: An empirical evaluation of convergence

On the Convergence of the Direct Extension of ADMM for Three-Block Separable Convex Minimization Models with One Strongly Convex Function.

A Hybrid Type ADMM for Multi-Block Separable Convex Programming

Convergence of ADMM for Multi-Block Nonconvex Separable Optimization Models

P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and Paralleled ADMM Approach

On the global and linear convergence of the generalized ADMM with three blocks

A Convergence Analysis of Nonlinearly Constrained ADMM in Deep Learning

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems