Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Achieving Linear Convergence of Distributed Optimization over Time Varying Directed Networks with Column Stochastic Weight Matrices

An Initialization-Free Continuous-Time Distributed Algorithm for Convex Optimization Problem

Distributed Stochastic Algorithm for Global Optimization in Networked System

Distributed Optimization of Multiagent Systems in Directed Networks with Time-Varying Delay.

Distributed optimization via multi-agent systems

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed Algorithm for a Finite Time Horizon Resource Allocation over a Directed Network

Distributed ADMM for Time-Varying Communication Networks

Distributed Heterogeneous Multi-Agent Optimization with Stochastic Sub-Gradient

An Expanded Distributed Algorithm for Dynamic Resource Allocation over Strongly Connected Topologies

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

A Continuous-Time Algorithm for Distributed Optimization With Nonuniform Time-Delay Under Switching and Unbalanced Digraphs

Linearly Convergent Second-Order Distributed Optimization Algorithms

A projection-based continuous-time algorithm for distributed optimization over multi-agent systems

Distributed Optimization in Prescribed-Time: Theory and Experiment

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Distributed Time-Varying Convex Optimization for a Class of Nonlinear Multiagent Systems

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

(Rectified Version) The Barzilai-Borwein Method for Distributed Optimization over Unbalanced Directed Networks

Delay-dependent LMI-based stability criterion for distributed optimization problem in heterogeneous linear multi-agent systems over random digraphs

On Inhomogeneous Infinite Products of Stochastic Matrices and Applications