Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Accelerated Alternating Direction Method of Multipliers: an Optimal O(1/K) Nonergodic Analysis

Faster and Non-ergodic O(1/K) Stochastic Alternating Direction Method of Multipliers.

Accelerated Stochastic ADMM with Variance Reduction

Zeroth-Order Online Alternating Direction Method of Multipliers: Convergence Analysis and Applications.

A Fast Symmetric Alternating Direction Method of Multipliers

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

O(1/t) Complexity Analysis of the Generalized Alternating Direction Method of Multipliers

On the O(1/k) Convergence and Parallelization of the Alternating Direction Method of Multipliers

A new accelerated positive-indefinite proximal admm for constrained separable convex optimization problems

Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration

Accelerated linearized alternating direction method of multipliers with Nesterov extrapolation

The Convergence Rate Analysis of the Symmetric ADMM for the Nonconvex Separable Optimization Problems

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

A general analysis of the convergence of ADMM

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

Parallel Multi-Block ADMM with O (1 / K ) Convergence

On the Global and Linear Convergence of the Generalized Alternating Direction Method of Multipliers

Alternating Direction Method of Multipliers with Difference of Convex Functions.

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints