Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Online Learning over Distributed Low-Rank Networks Via Sequential Power Iteration

Distributed Sparse Recursive Least-Squares over Networks

Fast and Robust Low-Rank Learning over Networks: A Decentralized Matrix Quantile Regression Approach

Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Distributed stochastic gradient descent for link prediction in signed social networks

A Distributed Conjugate Gradient Online Learning Method over Networks

Distributed Sparse Total Least-Squares over Networks

Distributed Linear Equations over Random Networks

Adaptation and learning over networks under subspace constraints -- Part I: Stability Analysis

A Distributed Gradient Algorithm Based on Randomized Block-Coordinate and Projection-Free over Networks

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks

Random Inverse Problems Over Graphs: Decentralized Online Learning

Proximal Multitask Learning Over Distributed Networks With Jointly Sparse Structure

Distributed Ordinal Regression Over Networks

Distributed adaptive Newton methods with global superlinear convergence

Efficient Stochastic Optimization for Low-Rank Distance Metric Learning.

Distributed Optimization With Personalization: A Flexible Algorithmic Framework

Online Convex Optimization over Erdos-Renyi Random Networks.

Distributed Constrained Nonlinear Least-Squares Estimation Algorithm Over Unbalanced Directed Networks

Decentralized projected Riemannian stochastic recursive momentum method for smooth optimization on compact submanifolds