Abstract:Alternating direction method of multipliers (ADMM) has received tremendous interest for solving numerous problems in machine learning, statistics and signal processing. However, it is known that the performance of ADMM and many of its variants is very sensitive to the penalty parameter of a quadratic penalty applied to the equality constraints. Although several approaches have been proposed for dynamically changing this parameter during the course of optimization, they do not yield theoretical improvement in the convergence rate and are not directly applicable to stochastic ADMM. In this paper, we develop a new ADMM and its linearized variant with a new adaptive scheme to update the penalty parameter. Our methods can be applied under both deterministic and stochastic optimization settings for structured non-smooth objective function. The novelty of the proposed scheme lies at that it is adaptive to a local sharpness property of the objective function, which marks the key difference from previous adaptive scheme that adjusts the penalty parameter per-iteration based on certain conditions on iterates. On theoretical side, given the local sharpness characterized by an exponent $\theta\in(0, 1]$, we show that the proposed ADMM enjoys an improved iteration complexity of $\widetilde O(1/\epsilon^{1-\theta})$\footnote{$\widetilde O()$ suppresses a logarithmic factor.} in the deterministic setting and an iteration complexity of $\widetilde O(1/\epsilon^{2(1-\theta)})$ in the stochastic setting without smoothness and strong convexity assumptions. The complexity in either setting improves that of the standard ADMM which only uses a fixed penalty parameter. On the practical side, we demonstrate that the proposed algorithms converge comparably to, if not much faster than, ADMM with a fine-tuned fixed penalty parameter.

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

Accelerated Stochastic ADMM with Variance Reduction

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

Convergence analysis of an inexact accelerated stochastic admm with larger stepsizes

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

Accelerated stochastic admm for empirical risk minimization

ADMM for Nonconvex Optimization under Minimal Continuity Assumption

Convergence on a Symmetric Accelerated Stochastic ADMMwith Larger Stepsizes

Adaptive smoothing mini-batch stochastic accelerated gradient method for nonsmooth convex stochastic composite optimization

An Efficient Inexact Symmetric Gauss-Seidel Based Majorized ADMM for High-Dimensional Convex Composite Conic Programming

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

A Unified Algorithmic Framework of Symmetric Gauss-Seidel Decomposition Based Proximal ADMMs for Convex Composite Programming

ADMM Without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization.

Dynamical convergence analysis for nonconvex linearized proximal ADMM algorithms

A partial Bregman ADMM with a general relaxation factor for structured nonconvex and nonsmooth optimization

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties