Abstract:Many problems in machine learning and other fields can be (re)formulated as linearly constrained separable convex programs. In most of the cases, there are multiple blocks of variables. However, the traditional alternating direction method (ADM) and its linearized version (LADM, obtained by linearizing the quadratic penalty term) are for the two-block case and cannot be naively generalized to solve the multi-block case. So there is great demand on extending the ADM based methods for the multi-block case. In this paper, we propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve multi-block separable convex programs efficiently. When all the component objective functions have bounded subgradients, we obtain convergence results that are stronger than those of ADM and LADM, e.g., allowing the penalty parameter to be unbounded and proving the sufficient and necessary conditions for global convergence. We further propose a simple optimality measure and reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with extra convex set constraints, with refined parameter estimation we devise a practical version of LADMPSAP for faster convergence. Finally, we generalize LADMPSAP to handle programs with more difficult objective functions by linearizing part of the objective function as well. LADMPSAP is particularly suitable for sparse representation and low-rank recovery problems because its subproblems have closed form solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration. It is also highly parallelizable and hence fits for parallel or distributed computing. Numerical experiments testify to the advantages of LADMPSAP in speed and numerical accuracy.

Alternating Direction Method for Separable Variables under Pair-Wise Constraints

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

Parallel Direction Method of Multipliers

Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Generalized Symmetric ADMM for Separable Convex Optimization

A Hybrid Type ADMM for Multi-Block Separable Convex Programming

The Direct Extension of ADMM for Multi-Block Convex Minimization Problems is Not Necessarily Convergent

A Note on the Sufficient Initial Condition Ensuring the Convergence of Directly Extended 3-Block ADMM for Special Semidefinite Programming

An Extended ADMM for 3-Block Nonconvex Nonseparable Problems with Applications

A New Alternating Direction Method for Solving Variational Inequalities

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

Online Alternating Direction Method (longer version)

A Generalization of Linearized Alternating Direction Method of Multipliers for Solving Two-Block Separable Convex Programming

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Two-block vs. Multi-block ADMM: An empirical evaluation of convergence

Inexact Alternating-Direction-Based Contraction Methods for Separable Linearly Constrained Convex Optimization

Multi-block relaxed-dual linear inertial ADMM algorithm for nonconvex and nonsmooth problems with nonseparable structures

Alternating Direction Augmented Lagrangian Methods for Semidefinite Programming

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses