Abstract:Many problems in machine learning and other fields can be (re)formulated as linearly constrained separable convex programs. In most of the cases, there are multiple blocks of variables. However, the traditional alternating direction method (ADM) and its linearized version (LADM, obtained by linearizing the quadratic penalty term) are for the two-block case and cannot be naively generalized to solve the multi-block case. So there is great demand on extending the ADM based methods for the multi-block case. In this paper, we propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve multi-block separable convex programs efficiently. When all the component objective functions have bounded subgradients, we obtain convergence results that are stronger than those of ADM and LADM, e.g., allowing the penalty parameter to be unbounded and proving the sufficient and necessary conditions for global convergence. We further propose a simple optimality measure and reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with extra convex set constraints, with refined parameter estimation we devise a practical version of LADMPSAP for faster convergence. Finally, we generalize LADMPSAP to handle programs with more difficult objective functions by linearizing part of the objective function as well. LADMPSAP is particularly suitable for sparse representation and low-rank recovery problems because its subproblems have closed form solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration. It is also highly parallelizable and hence fits for parallel or distributed computing. Numerical experiments testify to the advantages of LADMPSAP in speed and numerical accuracy.

On the Convergence of SARAH and Beyond

The convergence of conjugate gradient method with nonmonotone line search

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Stochastic asymptotical regularization for nonlinear ill-posed problems

A Modified Inexact SARAH Algorithm with Stabilized Barzilai-Borwein Step-Size in Machine learning

Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning

Convergence of LCA Flows to (C)LASSO Solutions

On the Convergence of L-shaped Algorithms for Two-Stage Stochastic Programming

On Convergence of Gradient Expected Sarsa($λ$)

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Stochastic asymptotical regularization for linear inverse problems

A constructive approach to L

Homotopy Based Algorithms for $\ell _{\scriptscriptstyle 0}$ -Regularized Least-Squares

Lock-free Optimization for Non-Convex Problems

NPASA: An algorithm for nonlinear programming -- Motivation and Global Convergence

Learning to optimize with convergence guarantees using nonlinear system theory

Finite-Time Decoupled Convergence in Nonlinear Two-Time-Scale Stochastic Approximation

Linear Convergence of the Alternating Direction Method of Multipliers for a Class of Convex Optimization Problems

Accelerating Mini-batch SARAH by Step Size Rules

On the Convergence of Loss and Uncertainty-based Active Learning Algorithms

$S^{2}$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning.