Abstract:Asynchronous-parallel algorithms have the potential to vastly speed up algorithms by eliminating costly synchronization. However, our understanding to these algorithms is limited because the current convergence of asynchronous (block) coordinate descent algorithms are based on somewhat unrealistic assumptions. In particular, the age of the shared optimization variables being used to update a block is assumed to be independent of the block being updated. Also, it is assumed that the updates are applied to randomly chosen blocks. In this paper, we argue that these assumptions either fail to hold or will imply less efficient implementations. We then prove the convergence of asynchronous-parallel block coordinate descent under more realistic assumptions, in particular, always without the independence assumption. The analysis permits both the deterministic (essentially) cyclic and random rules for block choices. Because a bound on the asynchronous delays may or may not be available, we establish convergence for both bounded delays and unbounded delays. The analysis also covers nonconvex, weakly convex, and strongly convex functions. We construct Lyapunov functions that directly model both objective progress and delays, so delays are not treated errors or noise. A continuous-time ODE is provided to explain the construction at a high level.

On the Convergence of Parallel Asynchronous Block-Iterative Computations

On the Convergence of the Accelerated Riccati Iteration Method.

A General Asynchronous Block Iterative Model with Related Convergence Conditions

On Parallel Multisplitting Block Iterative Methods for Linear Systems Arising in the Numerical Solution of Euler Equations.

Convergence Analysis of Asynchronous Stochastic Recursive Gradient Algorithms

Parallel Asynchronous Team Algorithms: Convergence and Performance Analysis

Convergence of generalized alternating iterative methods

Further Results on Convergence of Asynchronous Linear Iterations

Asynchronous Coordinate Descent under More Realistic Assumptions

Nonlinear Asynchronous Block Iterative Method

Parallel Block-iterative Methods: Application to the Load Flow Problem

Asynchronous block-iterative methods for almost linear equations

Convergence of the Cyclic and Quasi-cyclic Block Jacobi Methods

A Note on the Block Alternating Splitting Implicit Iteration Method for Complex Saddle‐point Problems

CONVERGENCE CONDITIONS FOR MATRIX MULTISPLITTING ITERATIVE METHODS

Asynchronous Parallel Computation and Its Applicability to Continuous-system Simulation

Convergence of the complex block Jacobi methods under the generalized serial pivot strategies

A unified analysis framework for iterative parallel-in-time algorithms

Formalization of Asymptotic Convergence for Stationary Iterative Methods

Practical Convergence of Distributed Stochastic Iterative Processes

Convergence of Batch Asynchronous Stochastic Approximation With Applications to Reinforcement Learning