Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Distributed Adaptive Subgradient Algorithms for Online Learning over Time-Varying Networks

Online Learning for DNN Training: A Stochastic Block Adaptive Gradient Algorithm

CADA: Communication-Adaptive Distributed Adam

Variance Reduced Diffusion Adaptation for Online Learning over Networks

Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

AdaDB: an Adaptive Gradient Method with Data-Dependent Bound.

An Adaptive Primal-Dual Subgradient Algorithm for Online Distributed Constrained Optimization

An Adaptive and Momental Bound Method for Stochastic Learning

On the Convergence of Decentralized Adaptive Gradient Methods

Projection-free Distributed Online Learning in Networks

Distributed Online Optimization in Time-Varying Unbalanced Networks Without Explicit Subgradients

Distributed Regularized Online Optimization Using Forward–backward Splitting

ABNGrad: adaptive step size gradient descent for optimizing neural networks

AdaXod: a new adaptive and momental bound algorithm for training deep neural networks

Distributed Adaptive Gradient Algorithm with Gradient Tracking for Stochastic Non-Convex Optimization

Quantized Adaptive Subgradient Algorithms and Their Applications

Faster Adaptive Decentralized Learning Algorithms

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Efficient Adaptive Online Learning Via Frequent Directions