Abstract:This paper studies a distributed multi-agent convex optimization problem. The system comprises multiple agents in this problem, each with a set of local data points and an associated local cost function. The agents are connected to a server, and there is no inter-agent communication. The agents' goal is to learn a parameter vector that optimizes the aggregate of their local costs without revealing their local data points. In principle, the agents can solve this problem by collaborating with the server using the traditional distributed gradient-descent method. However, when the aggregate cost is ill-conditioned, the gradient-descent method (i) requires a large number of iterations to converge, and (ii) is highly unstable against process noise. We propose an iterative pre-conditioning technique to mitigate the deleterious effects of the cost function's conditioning on the convergence rate of distributed gradient-descent. Unlike the conventional pre-conditioning techniques, the pre-conditioner matrix in our proposed technique updates iteratively to facilitate implementation on the distributed network. In the distributed setting, we provably show that the proposed algorithm converges linearly with an improved rate of convergence than the traditional and adaptive gradient-descent methods. Additionally, for the special case when the minimizer of the aggregate cost is unique, our algorithm converges superlinearly. We demonstrate our algorithm's superior performance compared to prominent distributed algorithms for solving real logistic regression problems and emulating neural network training via a noisy quadratic model, thereby signifying the proposed algorithm's efficiency for distributively solving non-convex optimization. Moreover, we empirically show that the proposed algorithm results in faster training without compromising the generalization performance.

MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methods

MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methods

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Augmented Distributed Gradient Methods for Multi-Agent Optimization under Uncoordinated Constant Stepsizes

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

A Unified Algorithmic Framework for Distributed Composite Optimization.

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Distributed Optimization Based on Gradient-tracking Revisited: Enhancing Convergence Rate via Surrogation

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

Accelerated Distributed Aggregative Optimization

Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

Achieving Acceleration in Distributed Optimization via Direct Discretization of the Heavy-Ball ODE

Accelerating Distributed Optimization via Fixed-time Convergent Flows: Extensions to Non-convex Functions and Consistent Discretization

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Multi-consensus Decentralized Accelerated Gradient Descent

Convergence of Asynchronous Distributed Gradient Methods over Stochastic Networks

The design of continuous-time accelerated algorithm for nonsmooth distributed optimization without sharing local decision variables

Combining Conjugate Gradient and Momentum for Unconstrained Stochastic Optimization With Applications to Machine Learning

Convergence of Distributed Adaptive Optimization with Local Updates

On Accelerating Distributed Convex Optimizations