Abstract:The alternating direction method of multipliers (ADMM) has been extensively used in a wide variety of different applications. When large datasets with high-dimensional variables are considered, subproblems arising from the ADMM must be inexactly solved even though they may theoretically have closed-form solutions. Such a scenario immediately poses mathematical ambiguities such as how these subproblems should be accurately solved and whether the convergence can still be guaranteed. Although the ADMM is well known, it seems that these topics should be deeply investigated. In this paper, we study the mathematics of how to implement the ADMM for a large dataset scenarios. More specifically, we attempt to focus on the convex programming case where there is a quadratic function with extremely high-dimensional variables in the objective function of the model; thereby there is a huge-scale system for linear equations needing to be solved at each iteration of the ADMM. It is revealed that there is no need, indeed it is impossible, to exactly solve this linear system, and we attempt to propose an adjustable inexactness criterion to automatically and inexactly solve this linear system. We further attempt to identify the safe-guard number for the internally nested iterations that can sufficiently ensure this inexactness criterion if the linear system would be solved by a standard numerical linear algebra solver. The convergence, together with the worst-case convergence rate measured by the iteration complexity, is rigorously established for the ADMM with inexactly solved subproblems. Some numerical experiments for large datasets of the least absolute shrinkage and selection operator containing millions of variables are reported to show the efficiency of the mentioned inaccurate implementation of the ADMM.

An Alternating Direction Method of Multipliers with the BFGS Update for Structured Convex Quadratic Optimization

Alternating Direction Method of Multipliers with Variable Metric Indefinite Proximal Terms for Convex Optimization

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

A Generalized Alternating Direction Method of Multipliers with Semi-Proximal Terms for Convex Composite Conic Programming

An Inertial Proximal Alternating Direction Method of Multipliers for Nonconvex Optimization.

A Golden Ratio Proximal Alternating Direction Method of Multipliers for Separable Convex Optimization

A Unified Algorithmic Framework of Symmetric Gauss-Seidel Decomposition Based Proximal ADMMs for Convex Composite Programming

A Unified Alternating Direction Method of Multipliers by Majorization Minimization

Stochastic linearized generalized alternating direction method of multipliers: Expected convergence rates and large deviation properties

BFGS-ADMM for Large-Scale Distributed Optimization

An Inertial Proximal Partially Symmetric ADMM-based Algorithm for Linearly Constrained Multi-Block Nonconvex Optimization Problems with Applications.

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

Convergence Analysis on a Modified Generalized Alternating Direction Method of Multipliers

An Extended Proximal ADMM Algorithm for Three-Block Nonconvex Optimization Problems

Adaptive stochastic alternating direction method of multipliers

The symmetric ADMM with indefinite proximal regularization and its application

A Hybrid Bregman Alternating Direction Method of Multipliers for the Linearly Constrained Difference-of-convex Problems

A Distributed Proximal Alternating Direction Multiplier Method for Multiblock Nonsmooth Composite Optimization

Fast Proximal Linearized Alternating Direction Method of Multiplier with Parallel Splitting

Implementing the Alternating Direction Method of Multipliers for Big Datasets: A Case Study of Least Absolute Shrinkage and Selection Operator