Abstract:This paper studies a distributed multi-agent convex optimization problem. The system comprises multiple agents in this problem, each with a set of local data points and an associated local cost function. The agents are connected to a server, and there is no inter-agent communication. The agents' goal is to learn a parameter vector that optimizes the aggregate of their local costs without revealing their local data points. In principle, the agents can solve this problem by collaborating with the server using the traditional distributed gradient-descent method. However, when the aggregate cost is ill-conditioned, the gradient-descent method (i) requires a large number of iterations to converge, and (ii) is highly unstable against process noise. We propose an iterative pre-conditioning technique to mitigate the deleterious effects of the cost function's conditioning on the convergence rate of distributed gradient-descent. Unlike the conventional pre-conditioning techniques, the pre-conditioner matrix in our proposed technique updates iteratively to facilitate implementation on the distributed network. In the distributed setting, we provably show that the proposed algorithm converges linearly with an improved rate of convergence than the traditional and adaptive gradient-descent methods. Additionally, for the special case when the minimizer of the aggregate cost is unique, our algorithm converges superlinearly. We demonstrate our algorithm's superior performance compared to prominent distributed algorithms for solving real logistic regression problems and emulating neural network training via a noisy quadratic model, thereby signifying the proposed algorithm's efficiency for distributively solving non-convex optimization. Moreover, we empirically show that the proposed algorithm results in faster training without compromising the generalization performance.

Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimization

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

Linear Convergence of Distributed Aggregative Optimization with Coupled Inequality Constraints

Linear Convergence of First- and Zeroth-Order Primal-Dual Algorithms for Distributed Nonconvex Optimization

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Primal-Dual Algorithm for Distributed Optimization with Coupled Constraints

Linear convergence of a primal-dual algorithm for distributed interval optimization

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Learning (With) Distributed Optimization

Accelerating Distributed Optimization: A Primal-Dual Perspective on Local Steps

Distributed Optimization Algorithm with Superlinear Convergence Rate

Distributed Zeroth-Order Optimization: Convergence Rates That Match Centralized Counterpart

Exponential Convergence of Augmented Primal-dual Gradient Algorithms for Partially Strongly Convex Functions

Distributed Dual Subgradient Methods with Averaging and Applications to Grid Optimization

Linear convergence of a dual optimization formulation for distributed optimization on directed graphs with unreliable communications

Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization

On Accelerating Distributed Convex Optimizations

A Smooth Double Proximal Primal-Dual Algorithm for a Class of Distributed Nonsmooth Optimization Problem

A Coordinate Descent Primal-Dual Algorithm and Application to Distributed Asynchronous Optimization