Abstract:Accompanied with the rising popularity of compressed sensing, the Alternating Direction Method of Multipliers (ADMM) has become the most widely used solver for linearly constrained convex problems with separable objectives. In this work, we observe that many existing ADMMs update the primal variable by minimizing different majorant functions with their convergence proofs given case by case. Inspired by the principle of majorization minimization, we respectively present the unified frameworks of Gauss-Seidel ADMMs and Jacobian ADMMs, which use different historical information for the current updating. Our frameworks generalize previous ADMMs to solve the problems with non-separable objectives. We also show that ADMMs converge faster when the used majorant function is tighter. We then propose the Mixed Gauss-Seidel and Jacobian ADMM (M-ADMM) which alleviates the slow convergence issue of Jacobian ADMMs by absorbing merits of the Gauss-Seidel ADMMs. M-ADMM can be further improved by backtracking and wise variable partition. We also propose to solve the multi-blocks problems by Proximal Gauss-Seidel ADMM which is of the Gauss-Seidel type. It convegences for non-strongly convex objective. Experiments on both synthesized and real-world data demonstrate the superiority of our new ADMMs. Finally, we release a toolbox that implements efficient ADMMs for many problems in compressed sensing.

Relaxing Alternating Direction Method of Multipliers (ADMM) for Linear Inverse Problems

Alternating Direction Method of Multipliers for Linear Inverse Problems

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Optimizing ADMM and Over-Relaxed ADMM Parameters for Linear Quadratic Problems

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

An alternating direction method of multipliers for inverse lithography problem

On convergence rates of proximal alternating direction method of multipliers

Multi-block relaxed-dual linear inertial ADMM algorithm for nonconvex and nonsmooth problems with nonseparable structures

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

An Empirical Study of ADMM for Nonconvex Problems

An adaptive linearized alternating direction multiplier method with a relaxation step for convex programming

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Sequential inertial linear ADMM algorithm for nonconvex and nonsmooth multiblock problems with nonseparable structure

A Riemannian ADMM

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

Understanding the ADMM Algorithm via High-Resolution Differential Equations

Accelerated Alternating Direction Method of Multipliers: an Optimal O(1/K) Nonergodic Analysis

Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration

Alternating direction method of multipliers for linear hyperspectral unmixing

Alternating Direction Method of Multipliers Based on $\ell_{2,0}$-norm for Multiple Measurement Vector Problem