Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Dynamical convergence analysis for nonconvex linearized proximal ADMM algorithms

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

A general analysis of the convergence of ADMM

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

On the Q-linear Convergence of a Majorized Proximal ADMM for Convex Composite Programming and Its Applications to Regularized Logistic Regression

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

A Unified Inexact Stochastic ADMM for Composite Nonconvex and Nonsmooth Optimization

Global Complexity Bound of a Proximal ADMM for Linearly Constrained Nonseparable Nonconvex Composite Programming

A Convergence Analysis of Nonlinearly Constrained ADMM in Deep Learning

A LINEARLY CONVERGENT MAJORIZED ADMM WITH INDEFINITE PROXIMAL TERMS FOR CONVEX COMPOSITE PROGRAMMING AND ITS APPLICATIONS

Differentiable Linearized ADMM

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

ADMM for Nonconvex Optimization under Minimal Continuity Assumption

Global Convergence of ADMM in Nonconvex Nonsmooth Optimization

Convergence analysis of an inexact accelerated stochastic admm with larger stepsizes

The Convergence Rate Analysis of the Symmetric ADMM for the Nonconvex Separable Optimization Problems

Convergence Analysis on a Modified Generalized Alternating Direction Method of Multipliers