Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Accelerated Design for Distributed Asynchronous Optimization

Fully asynchronous distributed optimization with linear convergence over directed networks

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

An Accelerated Algorithm for Distributed Optimization with Barzilai-Borwein Step Sizes

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

Convergence of an Accelerated Distributed Optimisation Algorithm over Time‐varying Directed Networks

Accelerated Distributed Aggregative Optimization

An Accelerated Decentralized Stochastic Optimization Algorithm with Inexact Model

Asynchronous Decentralized Accelerated Stochastic Gradient Descent

Asynchronous Decentralized Optimization in Directed Networks.

Asynchronous Distributed Optimization with Delay-free Parameters

Accelerated Distributed Nesterov Optimization Subject to Complex Constraints and Its Applications

Convergence of Asynchronous Distributed Gradient Methods over Stochastic Networks

Accelerated Convergence Algorithm for Distributed Constrained Optimization under Time-Varying General Directed Graphs.

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Nonsmooth Optimization

Asynchronous Subgradient-push Algorithm for Distributed Optimization over Directed Graphs

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Non-Smooth Optimization

Surplus-based accelerated algorithms for distributed optimization over directed networks

Accelerated $AB$/Push-Pull Methods for Distributed Optimization over Time-Varying Directed Networks

Achieving Linear Convergence in Distributed Aggregative Optimization over Directed Graphs