Abstract:In this paper, we establish the convergence of the proximal alternating direction method of multipliers (ADMM) and block coordinate descent (BCD) method for nonseparable minimization models with quadratic coupling terms. The novel convergence results presented in this paper answer several open questions that have been the subject of considerable discussion. We firstly extend the 2-block proximal ADMM to linearly constrained convex optimization with a coupled quadratic objective function, an area where theoretical understanding is currently lacking, and prove that the sequence generated by the proximal ADMM converges in point-wise manner to a primal-dual solution pair. Moreover, we apply randomly permuted ADMM (RPADMM) to nonseparable multi-block convex optimization, and prove its expected convergence for a class of nonseparable quadratic programming problems. When the linear constraint vanishes, the 2-block proximal ADMM and RPADMM reduce to the 2-block cyclic proximal BCD method and randomly permuted BCD (RPBCD). Our study provides the first iterate convergence result for 2-block cyclic proximal BCD without assuming the boundedness of the iterates. We also theoretically establish the expected iterate convergence result concerning multi-block RPBCD for convex quadratic optimization. In addition, we demonstrate that RPBCD may have a worse convergence rate than cyclic proximal BCD for 2-block convex quadratic minimization problems. Although the results on RPADMM and RPBCD are restricted to quadratic minimization models, they provide some interesting insights: (1) random permutation makes ADMM and BCD more robust for multi-block convex minimization problems; (2) cyclic BCD may outperform RPBCD for “nice” problems, and RPBCD should be applied with caution when solving general convex optimization problems especially with a few blocks.

On the Convergence of the Direct Extension of ADMM for Three-Block Separable Convex Minimization Models with One Strongly Convex Function.

Convergence of ADMM for Multi-Block Nonconvex Separable Optimization Models

The Direct Extension of ADMM for Multi-Block Convex Minimization Problems is Not Necessarily Convergent

On the global and linear convergence of the generalized ADMM with three blocks

A Convergent $3$-Block Semi-Proximal ADMM for Convex Minimization Problems with One Strongly Convex Block

Linearized Block-Wise Alternating Direction Method of Multipliers for Multiple-Block Convex Programming

On the Convergence Analysis of the Alternating Direction Method of Multipliers with Three Blocks

An Extended ADMM for 3-Block Nonconvex Nonseparable Problems with Applications

A Hybrid Type ADMM for Multi-Block Separable Convex Programming

Convergence of Multi-Block Bregman ADMM for Nonconvex Composite Problems

CONVERGENCE OF ADMM FOR OPTIMIZATION PROBLEMS WITH NONSEPARABLE NONCONVEX OBJECTIVE AND LINEAR CONSTRAINTS

A Note on the Sufficient Initial Condition Ensuring the Convergence of Directly Extended 3-Block ADMM for Special Semidefinite Programming

Linear Convergence Rate of Splitting Algorithms for Multi-Block Constrained Convex Minimizations

An Extended Proximal ADMM Algorithm for Three-Block Nonconvex Optimization Problems

Convergence Analysis of Generalized ADMM with Majorization for Linearly Constrained Composite Convex Optimization

A Bregman-style Partially Symmetric Alternating Direction Method of Multipliers for Nonconvex Multi-block Optimization

Convergent Prediction-Correction-based ADMM for Multi-Block Separable Convex Programming

Linearized ADMM for Nonconvex Nonsmooth Optimization with Convergence Analysis

The Convergence Rate Analysis of the Symmetric ADMM for the Nonconvex Separable Optimization Problems

A Partially Isochronous Splitting Algorithm for Three-Block Separable Convex Minimization Problems

Extended ADMM and BCD for Nonseparable Convex Minimization Models with Quadratic Coupling Terms: Convergence Analysis and Insights