Abstract:Accompanied with the rising popularity of compressed sensing, the Alternating Direction Method of Multipliers (ADMM) has become the most widely used solver for linearly constrained convex problems with separable objectives. In this work, we observe that many existing ADMMs update the primal variable by minimizing different majorant functions with their convergence proofs given case by case. Inspired by the principle of majorization minimization, we respectively present the unified frameworks of Gauss-Seidel ADMMs and Jacobian ADMMs, which use different historical information for the current updating. Our frameworks generalize previous ADMMs to solve the problems with non-separable objectives. We also show that ADMMs converge faster when the used majorant function is tighter. We then propose the Mixed Gauss-Seidel and Jacobian ADMM (M-ADMM) which alleviates the slow convergence issue of Jacobian ADMMs by absorbing merits of the Gauss-Seidel ADMMs. M-ADMM can be further improved by backtracking and wise variable partition. We also propose to solve the multi-blocks problems by Proximal Gauss-Seidel ADMM which is of the Gauss-Seidel type. It convegences for non-strongly convex objective. Experiments on both synthesized and real-world data demonstrate the superiority of our new ADMMs. Finally, we release a toolbox that implements efficient ADMMs for many problems in compressed sensing.

Local Linear Convergence of the Alternating Direction Method of Multipliers for Quadratic Programs

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

Local Convergence Properties of Douglas–Rachford and Alternating Direction Method of Multipliers

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

On convergence rates of proximal alternating direction method of multipliers

Local Linear Convergence of an ADMM-Type Splitting Framework for Equality Constrained Optimization

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

A LINEARLY CONVERGENT MAJORIZED ADMM WITH INDEFINITE PROXIMAL TERMS FOR CONVEX COMPOSITE PROGRAMMING AND ITS APPLICATIONS

A Fundamental Proof of Convergence of Alternating Direction Method of Multipliers for Weakly Convex Optimization

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Activity Identification and Local Linear Convergence of Douglas–Rachford/ADMM under Partial Smoothness

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

The symmetric ADMM with indefinite proximal regularization and its application

On the dual step length of the alternating direction method of multipliers

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Linearly Convergent Decentralized Consensus Optimization with the Alternating Direction Method of Multipliers

On the Linear Convergence of the ADMM in Decentralized Consensus Optimization