Abstract:This paper presents a novel accelerated distributed algorithm for unconstrained consensus optimization over static undirected networks. The proposed algorithm combines the benefits of acceleration from momentum, the robustness of the alternating direction method of multipliers, and the computational efficiency of gradient tracking to surpass existing state-of-the-art methods in convergence speed, while preserving their computational and communication cost. First, we prove that, by applying momentum on the average dynamic consensus protocol over the estimates and gradient, we can study the algorithm as an interconnection of two singularly perturbed systems: the outer system connects the consensus variables and the optimization variables, and the inner system connects the estimates of the optimum and the auxiliary optimization variables. Next, we prove that, by adding momentum to the auxiliary dynamics, our algorithm always achieves faster convergence than the achievable linear convergence rate for the non-accelerated alternating direction method of multipliers gradient tracking algorithm case. Through simulations, we numerically show that our accelerated algorithm surpasses the existing accelerated and non-accelerated distributed consensus first-order optimization protocols in convergence speed.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in the distributed algorithms for unconstrained consensus optimization on static undirected networks, how to combine momentum, alternating direction method of multipliers (ADMM) and gradient tracking to accelerate the convergence speed while keeping the computational and communication costs of the algorithm unchanged. Specifically, the paper proposes a new accelerated distributed optimization algorithm, aiming to outperform the existing state - of - the - art methods in terms of convergence speed. By applying momentum in the average dynamic consensus protocol of estimates and gradients, the algorithm can be regarded as the interconnection of two singularly perturbed systems: the outer system connects consensus variables and optimization variables, and the inner system connects optimal estimates and auxiliary optimization variables. By adding momentum to the auxiliary dynamics, the paper proves that the algorithm can always achieve a faster convergence speed than the non - accelerated ADMM - gradient - tracking algorithm. The main contributions of the paper are as follows: 1. **Accelerated Convergence**: By introducing momentum in the dynamics of auxiliary variables, the algorithm can overcome the limitations of the convergence rate of auxiliary variables in the existing ADMM - GT algorithms, thereby speeding up the overall convergence speed. 2. **Theoretical Analysis**: The paper proves the global exponential stability of the algorithm through the theory of singularly perturbed systems, and also proves that the addition of momentum can indeed accelerate convergence. 3. **Experimental Verification**: Through numerical simulation, the paper shows that the proposed accelerated algorithm is superior to the existing accelerated and non - accelerated distributed consensus first - order optimization protocols in terms of convergence speed, while maintaining the same computational - communication resource consumption, only increasing the memory burden. In conclusion, the goal of this paper is to improve the existing distributed optimization algorithms by introducing momentum, so that it can significantly improve the convergence speed while maintaining a lightweight computational and communication burden.

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

ADMM-Tracking Gradient for Distributed Optimization over Asynchronous and Unreliable Networks

An Accelerated Distributed Stochastic Gradient Method with Momentum

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Distributed Stochastic Consensus Optimization With Momentum for Nonconvex Nonsmooth Problems

Momentum-based Distributed Gradient Tracking Algorithms for Distributed Aggregative Optimization over Unbalanced Directed Graphs

Triggered Gradient Tracking for Asynchronous Distributed Optimization

Multi-consensus decentralized accelerated gradient descent

An Event-Triggered Approach for Gradient Tracking in Consensus-Based Distributed Optimization

Distributed Momentum Based Multi-Agent Optimization with Different Constraint Sets

A Nesterov-Like Gradient Tracking Algorithm for Distributed Optimization over Directed Networks

An Accelerated Gradient Tracking Algorithm with Projection Error for Distributed Optimization.

Accelerated $AB$/Push-Pull Methods for Distributed Optimization over Time-Varying Directed Networks

A Distributed Stochastic Optimization Algorithm with Gradient-Tracking and Distributed Heavy-Ball Acceleration

Distributed Linearized Alternating Direction Method of Multipliers for Composite Convex Consensus Optimization

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Convergence of an Accelerated Distributed Optimisation Algorithm over Time‐varying Directed Networks

Distributed Conjugate Gradient Method via Conjugate Direction Tracking

Explicit Convergence Rate of a Distributed Alternating Direction Method of Multipliers

Achieving Acceleration in Distributed Optimization via Direct Discretization of the Heavy-Ball ODE