Abstract:The alternating direction method of multipliers (ADMM) algorithm has been widely employed for distributed machine learning tasks. However, it suffers from several limitations, e.g., a relative low convergence speed, and an expensive time cost. To this end, in this paper, a novel method, namely the group-based ADMM (GADMM), is proposed for distributed linear classification. In particular, to accelerate the convergence speed and improve global consensus, a group layer is first utilized in GADMM to divide all the slave nodes into several groups. Then, all the local variables (from the slave nodes) are gathered in the group layer to generate different group variables. Finally, by using a weighted average method, the group variables are coordinated to update the global variable (from the master node) until the solution of the global problem is reached. According to the theoretical analysis, we found that: 1) GADMM can mathematically converge at the rate $O({1}/{k})$ , where ${k}$ is the number of outer iterations and 2) by using the grouping methods, GADMM can improve the convergence speed compared with the distributed ADMM framework without grouping methods. Moreover, we systematically evaluate GADMM on four publicly available LIBSVM datasets. Compared with disADMM and stochastic dual coordinate ascent with alternating direction method of multipliers-ADMM, for distributed classification, GADMM is able to reduce the number of outer iterations, which leads to faster convergence speed and better global consensus. In particular, the statistical significance test has been experimentally conducted and the results validate that GADMM can significantly save up to 30% of the total time cost (with less than 0.6% accuracy loss) compared with disADMM on large-scale datasets, e.g., webspam and epsilon.

A Generalized Formulation for Group Selection via ADMM

ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

An ADMM Algorithm for a Generic $\ell_0$ Sparse Overlapping Group Lasso Problem

A General Continuous-Time Formulation of Stochastic ADMM and Its Variants

An over-relaxed ADMM for separable convex programming and its applications to statistical learning

Generalized Symmetric ADMM for Separable Convex Optimization

A unified consensus-based parallel ADMM algorithm for high-dimensional regression with combined regularizations

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

A General Distributed Dual Coordinate Optimization Framework for Regularized Loss Minimization

ADMM for Nonsmooth Composite Optimization under Orthogonality Constraints

An Empirical Study of ADMM for Nonconvex Problems

Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification

A first-order optimization algorithm for statistical learning with hierarchical sparsity structure

A Block-wise, Asynchronous and Distributed ADMM Algorithm for General Form Consensus Optimization.

ADMM Algorithmic Regularization Paths for Sparse and Large Scale Positive-Definite Covariance Matrix Estimation

Understanding the ADMM Algorithm via High-Resolution Differential Equations

A Generalization of Linearized Alternating Direction Method of Multipliers for Solving Two-Block Separable Convex Programming

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

Laplacian Smoothing Stochastic ADMMs with Differential Privacy Guarantees

Consistent group selection in high-dimensional linear regression

Group Sparsity Extension of “ Non-Convex Sparse Regularization Via Convex Optimization for Impact Force Identification ”