Abstract:Accompanied with the rising popularity of compressed sensing, the Alternating Direction Method of Multipliers (ADMM) has become the most widely used solver for linearly constrained convex problems with separable objectives. In this work, we observe that many existing ADMMs update the primal variable by minimizing different majorant functions with their convergence proofs given case by case. Inspired by the principle of majorization minimization, we respectively present the unified frameworks of Gauss-Seidel ADMMs and Jacobian ADMMs, which use different historical information for the current updating. Our frameworks generalize previous ADMMs to solve the problems with non-separable objectives. We also show that ADMMs converge faster when the used majorant function is tighter. We then propose the Mixed Gauss-Seidel and Jacobian ADMM (M-ADMM) which alleviates the slow convergence issue of Jacobian ADMMs by absorbing merits of the Gauss-Seidel ADMMs. M-ADMM can be further improved by backtracking and wise variable partition. We also propose to solve the multi-blocks problems by Proximal Gauss-Seidel ADMM which is of the Gauss-Seidel type. It convegences for non-strongly convex objective. Experiments on both synthesized and real-world data demonstrate the superiority of our new ADMMs. Finally, we release a toolbox that implements efficient ADMMs for many problems in compressed sensing.

Alternating Direction Method of Multipliers with Difference of Convex Functions.

A Generalized Alternating Direction Method of Multipliers with Semi-Proximal Terms for Convex Composite Conic Programming

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

An inertial alternating direction method of multipliers for solving a two-block separable convex minimization problem

Modified Alternating Direction Method of Multipliers for Convex Quadratic Semidefinite Programming

A Linearized Alternating Direction Method of Multipliers with Substitution Procedure.

Dual Alternating Direction Method of Multipliers for Inverse Imaging

Parallel Direction Method of Multipliers

An Inertial Proximal Alternating Direction Method of Multipliers for Nonconvex Optimization.

Convergence Analysis on a Modified Generalized Alternating Direction Method of Multipliers

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

ADMM for Nonconvex Optimization under Minimal Continuity Assumption

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Alternating Direction Method of Multipliers for a Class of Nonconvex Bilinear Optimization: Convergence Analysis and Applications.

An Inexact Alternating Direction Method of Multipliers with Relative Error Criteria

A Hybrid Bregman Alternating Direction Method of Multipliers for the Linearly Constrained Difference-of-convex Problems

Iteratively Linearized Reweighted Alternating Direction Method of Multipliers for a Class of Nonconvex Problems

Convergence of Bregman alternating direction method with multipliers for nonconvex composite problems

Linear Rate Convergence of the Alternating Direction Method of Multipliers for Convex Composite Programming.

Alternating Direction Method of Multipliers for Linear Inverse Problems

Convergence of alternating direction method for minimizing sum of two nonconvex functions with linear constraints