Abstract:Many problems in machine learning and other fields can be (re)formulated as linearly constrained separable convex programs. In most of the cases, there are multiple blocks of variables. However, the traditional alternating direction method (ADM) and its linearized version (LADM, obtained by linearizing the quadratic penalty term) are for the two-block case and cannot be naively generalized to solve the multi-block case. So there is great demand on extending the ADM based methods for the multi-block case. In this paper, we propose LADM with parallel splitting and adaptive penalty (LADMPSAP) to solve multi-block separable convex programs efficiently. When all the component objective functions have bounded subgradients, we obtain convergence results that are stronger than those of ADM and LADM, e.g., allowing the penalty parameter to be unbounded and proving the sufficient and necessary conditions for global convergence. We further propose a simple optimality measure and reveal the convergence rate of LADMPSAP in an ergodic sense. For programs with extra convex set constraints, with refined parameter estimation we devise a practical version of LADMPSAP for faster convergence. Finally, we generalize LADMPSAP to handle programs with more difficult objective functions by linearizing part of the objective function as well. LADMPSAP is particularly suitable for sparse representation and low-rank recovery problems because its subproblems have closed form solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration. It is also highly parallelizable and hence fits for parallel or distributed computing. Numerical experiments testify to the advantages of LADMPSAP in speed and numerical accuracy.

On Partial Smoothness, Activity Identification and Faster Algorithms of L1 over L2 Minimization

On Partly Smoothness, Activity Identification and Faster Algorithms of L 1 over L 2 Minimization

On Partly Smoothness, Activity Identification and Faster Algorithms of $L_1$ over $L_2$ Minimization

Smoothed sparse recovery via locally competitive algorithm and forward Euler discretization method.

On the Convergence of an Active-Set Method for ℓ1minimization

A Unified Study on $L_1$ over $L_2$ Minimization

Study on L_1 over L_2 Minimization for Nonnegative Signal Recovery

Study on <i>L</i><sub>1</sub> over <i>L</i><sub>2</sub> Minimization for Nonnegative Signal Recovery

Unified Analysis on L1 over L2 Minimization for Signal Recovery

A Learned Proximal Alternating Minimization Algorithm and Its Induced Network for a Class of Two-block Nonconvex and Nonsmooth Optimization

Nonlinear Eigen-approach ADMM for Sparse Optimization on Stiefel Manifold

An efficient active-set method with applications to sparse approximations and risk minimization

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation

Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation.

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

A Riemannian ADMM

Smoothed Proximal Lagrangian Method for Nonlinear Constrained Programs

Nonconvex Mixed Matrix Minimization

Linearized Alternating Direction Method with Parallel Splitting and Adaptive Penalty for Separable Convex Programs in Machine Learning