Abstract:Distributed stochastic non-convex optimization problems have recently received attention due to the growing interest of signal processing, computer vision, and natural language processing communities in applications deployed over distributed learning systems (e.g., federated learning). We study the setting where the data is distributed across the nodes of a time-varying directed network, a topology suitable for modeling dynamic networks experiencing communication delays and straggler effects. The network nodes, which can access only their local objectives and query a stochastic first-order oracle to obtain gradient estimates, collaborate to minimize a global objective function by exchanging messages with their neighbors. We propose an algorithm, novel to this setting, that leverages stochastic gradient descent with momentum and gradient tracking to solve distributed non-convex optimization problems over time-varying networks. To analyze the algorithm, we tackle the challenges that arise when analyzing dynamic network systems which communicate gradient acceleration components. We prove that the algorithm's oracle complexity is $\mathcal{O}(1/\epsilon^{1.5})$, and that under Polyak-$Ł$ojasiewicz condition the algorithm converges linearly to a steady error state. The proposed scheme is tested on several learning tasks: a non-convex logistic regression experiment on the MNIST dataset, an image classification task on the CIFAR-10 dataset, and an NLP classification test on the IMDB dataset. We further present numerical simulations with an objective that satisfies the PL condition. The results demonstrate superior performance of the proposed framework compared to the existing related methods.

Distributed Optimization with Binary Relative Information over Deterministically Time-varying Graphs

A Distributed Algorithm for Resource Allocation Over Dynamic Digraphs.

Distributed Discrete-Time Optimization by Exchanging One Bit of Information

Distributed Optimization For Resource Allocation With Event-Triggered Communication Under Directed Topology

Distributed Stochastic Algorithm for Global Optimization in Networked System

Distributed Optimization in Multi-agent Networks Using One-bit of Relative State Information

Distributed Stochastic Optimization With Unbounded Subgradients Over Randomly Time-Varying Networks

Distributed Convex Optimization with Inequality Constraints over Time-Varying Unbalanced Digraphs.

An Expanded Distributed Algorithm for Dynamic Resource Allocation over Strongly Connected Topologies

Distributed algorithm for solving variational inequalities over time-varying unbalanced digraphs

Discretized Distributed Optimization over Dynamic Digraphs

Distributed Discrete-Time Optimization in Multiagent Networks Using Only Sign of Relative State

Distributed Online Optimization in Time-Varying Unbalanced Networks Without Explicit Subgradients

Distributed Continuous‐time Constrained Convex Optimization with General Time‐varying Cost Functions

Distributed Discrete-time Optimization in Multi-agent Networks Using only Sign of Relative State

Distributed Adaptive Convex Optimization on Directed Graphs Via Continuous-Time Algorithms

Privacy-Preserving Distributed Online Optimization Over Unbalanced Digraphs via Subgradient Rescaling

Distributed Algorithm for a Finite Time Horizon Resource Allocation over a Directed Network

Distributed Continuous-Time Algorithms for Time-Varying Constrained Convex Optimization

Distributed quasi-monotone subgradient algorithm for nonsmooth convex optimization over directed graphs

Accelerated Distributed Stochastic Non-Convex Optimization over Time-Varying Directed Networks