Abstract:For a symmetric positive semidefinite linear system of equations $\mathcal{Q} {\bf x} = {\bf b}$, where ${\bf x} = (x_1,\ldots,x_s)$ is partitioned into $s$ blocks, with $s \geq 2$, we show that each cycle of the classical block symmetric Gauss-Seidel (block sGS) method exactly solves the associated quadratic programming (QP) problem but added with an extra proximal term of the form $\frac{1}{2} \| {\bf x}-{\bf x}^k \|_{\mathcal T}^2$, where ${\mathcal T}$ is a symmetric positive semidefinite matrix related to the sGS decomposition and ${\bf x}^k$ is the previous iterate. By leveraging on such a connection to optimization, we are able to extend the result (which we name as the block sGS decomposition theorem) for solving a convex composite QP (CCQP) with an additional possibly nonsmooth term in $x_1$, i.e., $\min\{ p(x_1) + \frac{1}{2}\langle {\bf x},\, \mathcal{Q} {\bf x} \rangle -\langle {\bf b},\, {\bf x}\rangle\}$, where $p(\cdot)$ is a proper closed convex function. Based on the block sGS decomposition theorem, we are able to extend the classical block sGS method to solve a CCQP. In addition, our extended block sGS method has the flexibility of allowing for inexact computation in each step of the block sGS cycle. At the same time, we can also accelerate the inexact block sGS method to achieve an iteration complexity of $O(1/k^2)$ after performing $k$ block sGS cycles. As a {fundamental} building block, the block sGS decomposition theorem has played a key role in various recently developed algorithms such as the inexact semiproximal {ALM/ADMM} for linearly constrained multi-block convex composite conic programming (CCCP), and the accelerated block coordinate descent method for multi-block CCCP.

Convergence analysis of a block improvement method for polynomial optimization over unit spheres

Convergence of hybrid polynomial approximation of rational Bézier curves

On Convergence of the Maximum Block Improvement Method

Extended Newton Methods for Multiobjective Optimization: Majorizing Function Technique and Convergence Analysis

Tighter bound estimation for efficient biquadratic optimization over unit spheres

A Bregman-Style Improved ADMM and its Linearized Version in the Nonconvex Setting: Convergence and Rate Analyses

An Improved Quadrature Scheme in B-spline Material Point Method for Large-Deformation Problem Analysis

Block majorization-minimization with diminishing radius for constrained nonconvex optimization

Convergence Behavior of Gauss-Newton's Method and Extensions of the Smale Point Estimate Theory.

Convergence and complexity of block majorization-minimization for constrained block-Riemannian optimization

A Unified Convergence Analysis of a Second-Order Method of Multipliers for Nonlinear Conic Programming

Flexible block-iterative analysis for the Frank-Wolfe algorithm

On the Geometric Analysis of a Quartic–quadratic Optimization Problem under a Spherical Constraint

Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

On convergence of the block Lanczos method for the CDT subproblem

Blockwise Stochastic Variance-Reduced Methods with Parallel Speedup for Multi-Block Bilevel Optimization

A block symmetric Gauss-Seidel decomposition theorem for convex composite quadratic programming and its applications

Improved Analyses of the Randomized Power Method and Block Lanczos Method

Quadratic Growth and Linear Convergence of a DCA Method for Quartic Minimization over the Sphere

Convergence analysis of a stochastic heavy-ball method for linear ill-posed problems