Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

Multi-stage stochastic gradient method with momentum acceleration

Accelerated Stochastic ADMM with Variance Reduction

Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Accelerated stochastic admm for empirical risk minimization

Adaptive momentum with discriminative weight for neural network stochastic optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Accelerated Learning for Restricted Boltzmann Machine with a Novel Momentum Algorithm

Stochastic Gradient Descent with Nonlinear Conjugate Gradient-Style Adaptive Momentum

Accelerated Variance Reduced Stochastic ADMM

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence

DEAM: Adaptive Momentum with Discriminative Weight for Stochastic Optimization

Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum

Convergence and Stability of the Stochastic Proximal Point Algorithm with Momentum

Optimal Adaptive and Accelerated Stochastic Gradient Descent

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

An Improved Analysis of Stochastic Gradient Descent with Momentum

ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent

Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning