Abstract:Alternating direction method of multipliers (ADMM) has received tremendous interests for solving numerous problems in machine learning, statistics and signal processing. However, it is well-known that the performance of ADMM and many of its variants are very sensitive to the penalty parameter of the quadratic term for the equality constraint. Although some useful heuristic approaches have been proposed for dynamically changing the penalty parameter during the course of optimization, none of them have been shown to yield {it explicit improvement} in the convergence rate and are appropriate for stochastic ADMM. It remains an open problem how to establish explicitly faster convergence of ADMMs by leveraging an adaptive scheme of penalty parameters. In this paper, we develop a new theory of (linearized) ADMMs with a new adaptive scheme of the penalty parameter for both {it deterministic and stochastic} optimization problems with non-smooth structured regularizers. The novelty of the proposed adaptive penalty scheme of lies at it is adaptive to a local sharpness property of the objective function, which also marks the key difference from previous work that focus on self-adaptivity in deterministic optimization by adjusting the penalty parameter per iteration based on the iterate message. On theoretical side, with the local sharpness characterized by a constant $thetain(0, 1)$, we show that the proposed deterministic ADMM enjoys an improved iteration complexity of $widetilde O(1/epsilon^{1-theta})$footnote{$widetilde O()$ suppresses a logarithmic factor.}, and the proposed stochastic ADMM enjoys an iteration complexity of $widetilde O(1/epsilon^{2(1-theta)})$ without smoothness and strong convexity assumptions, both of which improve that of their standard counterparts with a constant penalty parameter. On practical side, we demonstrate the proposed algorithms converge comparably if not faster than ADMM with a fine-tuned fixed penalty parameter.

Self-adaptive ADMM for semi-strongly convex problems

ADMM Without a Fixed Penalty Parameter: Faster Convergence with New Adaptive Penalization.

No More Fixed Penalty Parameter in ADMM: Faster Convergence with New Adaptive Penalization

Accelerated Stochastic ADMM with Variance Reduction

An Adaptive Proximal ADMM for Nonconvex Linearly-Constrained Composite Programs

Accelerated Nonconvex ADMM with Self-Adaptive Penalty for Rank-Constrained Model Identification

An Empirical Study of ADMM for Nonconvex Problems

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

Accelerating preconditioned ADMM via degenerate proximal point mappings

A Corrected Semi-Proximal ADMM for Multi-Block Convex Optimization and Its Application to DNN-SDPs

CONVERGENCE OF ADMM FOR OPTIMIZATION PROBLEMS WITH NONSEPARABLE NONCONVEX OBJECTIVE AND LINEAR CONSTRAINTS

Parallelized ADMM with General Objectives for Deep Learning.

On the Convergence of ADMM with Task Adaption and Beyond

Accelerating ADMM for Efficient Simulation and Optimization

The symmetric ADMM with indefinite proximal regularization and its application

A Convergent $3$-Block Semi-Proximal ADMM for Convex Minimization Problems with One Strongly Convex Block

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

Proximal ADMM for Nonconvex and Nonsmooth Optimization

An Efficient ADMM-Based Algorithm to Nonconvex Penalized Support Vector Machines.

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization