Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

An Empirical Study of ADMM for Nonconvex Problems

Adaptive Relaxed ADMM: Convergence Theory and Practical Implementation

Accelerated Stochastic ADMM with Variance Reduction

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

A Riemannian ADMM

ADMM for Nonconvex Optimization under Minimal Continuity Assumption

Optimizing ADMM and Over-Relaxed ADMM Parameters for Linear Quadratic Problems

Two-block vs. Multi-block ADMM: An empirical evaluation of convergence

Understanding the ADMM Algorithm via High-Resolution Differential Equations

Alternating Direction Method of Multipliers for Linear Inverse Problems

Convergence for nonconvex ADMM, with applications to CT imaging

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Optimal parameter selection for the alternating direction method of multipliers (ADMM): quadratic problems

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

An Adaptive Proximal ADMM for Nonconvex Linearly-Constrained Composite Programs

Relaxing Alternating Direction Method of Multipliers (ADMM) for Linear Inverse Problems

Sequential inertial linear ADMM algorithm for nonconvex and nonsmooth multiblock problems with nonseparable structure

An Extended ADMM for 3-Block Nonconvex Nonseparable Problems with Applications

An over-relaxed ADMM for separable convex programming and its applications to statistical learning